Emissions by louispt1 · Pull Request #1754 · quintel/etengine

louispt1 · 2026-05-21T18:50:11Z

Summary of changes

Emissions Calculation Framework

New DirectEmissions module implementing mass-balance equations for fossil and biogenic CO2
Tracks carbon flows: input content (A) + utilization (B) - output content (C) - capture (D) = emissions (E)
New MoleculeEmissions module for emissions reporting on molecule nodes
Support for CO2 capture (via ccs_capture_rate) and utilization (via co2_utilisation_per_mj)

Carbon Content Tracing

RecursiveFactor::DirectEmissions module for tracing CO2 content through supply chains
Handles mixed carriers (network gas, crude oil) by recursively calculating weighted composition
New emissions_skip_crude_oil_mix edge group for forcing weighted mix calculations

GQL & Dataset Integration

New EMISSIONS() GQL function for accessing emissions data from datasets
Emissions data loading infrastructure in Dataset::Import and Etsource::Loader
Graph-level emissions hash storage

Supporting Infrastructure

emissions node group for nodes participating in emissions tracking
ccus_captured node group for CO2 removal/capture technologies
- Uses lazy memoization for ccus_captured? check to avoid Marshal serialization issues
Updated biogenic emissions (primary) to use free_co2_factor for capture calculations with consistent results

Other

Reporting methods return nil for non-emissions nodes (checked via with_emissions_node)
Atlas gem updated to support emissions data structures

Note: Atlas reference should be updated once the Atlas emissions branch has been merged.
Goes with:

kndehaan · 2026-05-22T11:05:49Z

Note that direct_co2_output_content_carriers_biogenic doesn't give the correct results yet, also leading to incorrect results for methods that depend on this one.

This has still to with the potential_co2_conversion_per_mj attribute missing on carriers, as a consequence going to recursion where it should not always be so. This is something that we still need to look into.

louispt1 · 2026-05-27T08:39:10Z

@kndehaan I have pushed a fix for the biogenic output content carriers now

noracato

I'm worried that the UPDATE part does not have enough validation.

Can you please add tests that show updating nonexistent keys will not work? I have a feeling we will need the DatasetAttributes module after all (it is still on the emissions-gql branch)

noracato · 2026-05-28T17:27:38Z

    # See Qernel::Dataset#assign_dataset_attributes to understand what's going on:
    call_on_each_qernel_object(:assign_dataset_attributes)
+    # Manually assign emissions hash (not a DatasetAttributes object)
+    @emissions = dataset&.data&.[](:emissions) || {}


Can you explain once more why not to use DatasetAttributes? Like this there is no validation on what users can set as an emissions attribute.

Good point, I was not really looking at UPDATE yet, so I thought to keep it as light as possible for read.
However, you are right that it's cleaner to just use the DatasetAttributes approach for consistency and validation.

I re-instated the DatasetAttributes approach and added more tests to cover the case of nonexistent keys etc. I expect more spec will be needed based on the changes for 1990 but I think it's best to process that separately.

kndehaan · 2026-06-01T12:37:57Z

Adding myself as reviewer to check the descriptions for the methods (it should be understandable and correct from a modeller's perspective as well).

kndehaan

I put up some text suggestions and placed one comment about removing a method. Let me know if you have questions @louispt1

kndehaan · 2026-06-01T13:26:38Z

+      # Total CO2 utilised (consumed as feedstock) at this node.
+      #
+      # Currently returns only fossil utilisation, as biogenic utilisation is always 0.
+      #
+      # @return [Float, nil] Total CO2 utilised in kg, or nil if node is not in emissions group
+      def direct_co2_input_utilisation
+        with_emissions_node do
+          direct_co2_input_utilisation_fossil
+          # Potentially in the future: + direct_co2_input_utilisation_biogenic (currently 0)
+        end
+      end


I think we decided to remove this method for now, as we currently don't use it right now, right? @louispt1

Yes this method snuck back in with the csv serialiser merge - I will remove it

noracato · 2026-06-02T06:24:57Z

+      VALID_GHG_TYPES = %w[co2 other_ghg n2o ch4 hfc pfc sf6 nf3].freeze
+      VALID_GHG_PATTERN = /^(#{VALID_GHG_TYPES.join('|')})(_\d{4})?$/.freeze


Sorry to comment again, but let's not use constants with carriers and patterns. These keys have nothing to do with engine validations, they are part of the dataset pipeline IMO - or at least ETSource. The engine should not carry this knowledge.

No need to apologise, you're spot on. I pushed a change removing the validation.
Edit:
However now I need to figure out how to handle validation a little smarter - still working on it :)

@noracato what do you think about this approach?

noracato

Sorry to keep nitpicking. There is still logic present in the module that I feel over complicates it.

noracato · 2026-06-02T10:24:51Z

+        # For setters, check if the emission key exists in the dataset
+        if method_name.to_s.end_with?('=')
+          data_key = scoped_method(method_name.to_s.sub(/=$/, ''))
+          @emissions.dataset_has_key?(data_key)
+        else
+          # Getters always respond (may return nil if key doesn't exist)
+          true
+        end


Suggested change

# For setters, check if the emission key exists in the dataset

if method_name.to_s.end_with?('=')

data_key = scoped_method(method_name.to_s.sub(/=$/, ''))

@emissions.dataset_has_key?(data_key)

else

# Getters always respond (may return nil if key doesn't exist)

true

end

# Getters always respond (may return nil if key doesn't exist)

return true unless method_name.to_s.end_with?('=')

@emissions.dataset_has_key?(

scoped_method(method_name.to_s.sub(/=$/, ''))

)

General approach is good. This is a bit pythonesque ;)

Actually, what was wrong with:

def respond_to_missing?(method_name, include_private = false) data_key = scoped_method(method_name).split('=').first @emissions.respond_to?(data_key) || super end

noracato · 2026-06-02T10:26:26Z

+      # Check both string and symbol keys since datasets may use either
+      dataset_attributes.key?(key.to_s) || dataset_attributes.key?(key.to_sym)


I don't feel this should be handled here?

kndehaan

I found a critical bug that needs to be fixed. It concerns using the EMISSIONS() function for setting demand on dataset value molecule nodes. It works for the nl datasets (ETDataset datasets), but I think it does not work as it should for ETLocal datasets.

Example blank nl2023 scenario, querying the three things below gives the expected, desired other ghg emissions (don't mind the differences in decimals):

EACH(
  EMISSIONS(agriculture_non_specified, non_energetic, other_ghg),
  MV(agriculture_non_specified_non_energetic_other_ghg, demand),
  MV(agriculture_non_specified_non_energetic_other_ghg, direct_reporting_emissions_other_ghg_emissions)
)

[
  17,888.11512,
  17,888,115,120.0,
  17,888,115,120.0,
]

This is however not the case for for example blank NO_norway scenario (and other countries as well):

EACH(
  EMISSIONS(agriculture_non_specified, non_energetic, other_ghg),
  MV(agriculture_non_specified_non_energetic_other_ghg, demand),
  MV(agriculture_non_specified_non_energetic_other_ghg, direct_reporting_emissions_other_ghg_emissions)
)



[
  4,516.46047,
  0.0,
  0.0,
]

So I think something in reading with ~ demand on the molecule nodes goes wrong when it's a ETLocal dataset (derived dataset).

I know that for energy nodes, the derived dataset first looks if graph_methods is specified. If not, it falls back to what is defined with ~, whereas full datasets directly look at ~ and don't look at graph_methods.

Might it be that for the molecule nodes, something has to be configured for derived datasets to also look at values specified with~?

noracato · 2026-06-11T17:25:25Z

I'll have a look. Gut feeling is that you are right. There might also be a hidden scaling factor involved.

kndehaan · 2026-06-11T17:26:51Z

Additionally, if I set a hard-coded value on the molecule node like ~ demand = 1000.0, querying the demand for this node for Norway returns zero, whereas it does return the hard-coded value for nl2023. This confirms that something doesn't go right for setting demand with ~ method for derived datasets.

noracato · 2026-06-11T18:55:40Z

I found the culprit.

When calculating the initial graph, there is an extra thing applied for Derived datasets (ETLocal datasets). It is called ZeroMoleculeNodes. And it just sets the demand for all molecule nodes to zero.

Here you can see a comment saying "Derived datasets have no molecule flows."

I will try to find out why this was setup like this, and if we can lift it, or if we can create a special case for "floating" nodes.

mabijkerk · 2026-06-16T12:45:25Z

In the end, working with the emission methods as they are defined now is not the most intuitive: it is a bit too elaborate in practice. How did you experience this @kndehaan? Not saying this is a must have, but still good to flag.

More straightforward might be:

current method label	proposed method label
direct_co2_input_content_carriers_fossil	direct_co2_content_input_fossil
direct_co2_input_utilisation_fossil	direct_co2_utilisation_fossil
direct_co2_output_content_carriers_fossil	direct_co2_content_output_fossil
n.a.	direct_co2_production_fossil
direct_co2_output_production_capture_fossil	direct_co2_capture_fossil
direct_co2_output_production_emissions_fossil	direct_co2_emissions_fossil

Also, a direct_reporting_emissions_co2_emissions method would be useful. It is not the most intuitive that I have to use the direct_reporting_emissions_total_ghg_emissions method to get the complete picture of a node.

kndehaan · 2026-06-18T06:35:32Z

@mabijkerk let's review your comment in the next increment. I agree that the naming of the methods has not been most intuitive. Also, being able to query the direct reporting CO2 emissions of a node would be useful.

Dismissing the requested changes as the required solution has been implemented via Atlas, Transformer, ETSource.

…ion methods

…ions

…co2_output_production_emissions_fossil and spec

…ce of ccs_capture_rate. Update spec

…thod as direct_co2_input_utilisation_fossil is sufficient

…_utilised and emissions_lulucf_removals checks from spec

…f required

…now it covers more cases

…cus_captured?

…ut co2 content approach

…ents

…spec

* Expand ConfiguredCSVSerializer with node group expansion, molecule support, and emissions CSV endpoints * Add ghg_carrier method to DirectEmissions/MoleculeEmissions and update test fixture of the direct_emissions_csv * Add ghg_carrier to MoleculeEmissions specs and simplify DirectEmissions specs to use faster mock-based approach * Return symbols from ghg_carrier instead of strings

Co-authored-by: kndehaan <102598197+kndehaan@users.noreply.github.com>

… the emissions.csv file

…gql approach more closely

* Add a validation lib spec for node values per dataset * Add dev and test modes for graph data validation

louispt1 requested a review from noracato May 21, 2026 18:50

louispt1 force-pushed the emissions branch from 06c23dd to d423987 Compare May 27, 2026 14:58

louispt1 marked this pull request as ready for review May 27, 2026 14:59

noracato requested changes May 28, 2026

View reviewed changes

This was referenced May 29, 2026

Add direct emissions method with data export quintel/etsource#3444

Merged

Add direct emissions method with data export quintel/etmodel#4719

Merged

Emissions quintel/atlas#189

Merged

louispt1 force-pushed the emissions branch from f4e3149 to f136c00 Compare June 1, 2026 07:38

kndehaan self-requested a review June 1, 2026 12:37

kndehaan reviewed Jun 1, 2026

View reviewed changes

noracato requested changes Jun 2, 2026

View reviewed changes

kndehaan mentioned this pull request Jun 3, 2026

Add direct emissions method with data export quintel/documentation#321

Merged

3 tasks

noracato approved these changes Jun 4, 2026

View reviewed changes

kndehaan previously requested changes Jun 11, 2026

View reviewed changes

louispt1 added 6 commits June 19, 2026 10:57

Implement A, C and E: input and output without capture, & CO2 product…

9698084

…ion methods

Add guard based on the emissions group

4133474

Apply recursive edge logic to output and input edges for direct emiss…

3415517

…ions

Dataset EMISSIONS method implementation

48f9efd

Add emissions methods for molecule nodes

38b3711

Improve crude oil tests and recursion through multiple levels

c9a9ca1

louispt1 and others added 29 commits June 19, 2026 10:57

Refactor direct emissions, introduce capture methods, correct direct_…

796b09a

…co2_output_production_emissions_fossil and spec

Update existing engine methods to account for captured CO2

df72f84

Update bio-emissions to use free_co2_factor, depending on the prescen…

7f0f994

…ce of ccs_capture_rate. Update spec

Update methods based on excel formulas

ef42079

Adjust comments, add TODOs and remove direct_co2_input_utilisation me…

efd6ec8

…thod as direct_co2_input_utilisation_fossil is sufficient

Implement emissions method for CO2 capture on molecule nodes

d13bbf7

Handle co2_per_mj check when recursing for bio emissions, remove ccus…

def3c5f

…_utilised and emissions_lulucf_removals checks from spec

Update atlas reference to explicit commit hash

a997dfe

Removed scoped accessor in favour of hash - can re-instate accessor i…

df60994

…f required

return ccus_captured to eager memoization to match other methods and …

027d592

…now it covers more cases

Resolve biogenic recursion issue and return to lazy memoization for c…

b205e7c

…cus_captured?

Node colour for Waste and LULUCF sectors

db38dcb

Fix for biogenic output

f7247b2

Add node level check for biogenic CO2, simplify direct emissions outp…

b56e2bb

…ut co2 content approach

Clean up comments for PR

19e29d4

Update atlas reference to latest commit hash

903ba1c

Replace ABCDE references with worded fomulas in direct emissions comm…

faa067f

…ents

Restore DatasetAttribute approach and add additional UPDATE tests

2b067c5

Fix bug for crude oil edge case biogenic direct emissions and expand …

908a57c

…spec

Apply suggestions to modeller comments direct emissions methods

aa799fc

Co-authored-by: kndehaan <102598197+kndehaan@users.noreply.github.com>

Remove direct_co2_input_utilisation method

6b4d112

Remove hardcoded validations from Qernel::Emissions

382c288

Validation relies on the set of emissions keys and types available in…

db50d84

… the emissions.csv file

Revert to strict accessor and update specs to match, mimic emissions-…

42b6458

…gql approach more closely

Emission validation (#1759)

37d6950

* Add a validation lib spec for node values per dataset * Add dev and test modes for graph data validation

Bump atlas commit

9a52043

Bump Atlas to quintel/atlas@db94f40

d66d89e

Bump Atlas to quintel/atlas@21a6046

7404b0d

kndehaan force-pushed the emissions branch from 85cd750 to 7404b0d Compare June 19, 2026 08:57

		VALID_GHG_TYPES = %w[co2 other_ghg n2o ch4 hfc pfc sf6 nf3].freeze
		VALID_GHG_PATTERN = /^(#{VALID_GHG_TYPES.join('\|')})(_\d{4})?$/.freeze

		# Check both string and symbol keys since datasets may use either
		dataset_attributes.key?(key.to_s) \|\| dataset_attributes.key?(key.to_sym)

Conversation

louispt1 commented May 21, 2026 • edited by kndehaan Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Emissions Calculation Framework

Carbon Content Tracing

GQL & Dataset Integration

Supporting Infrastructure

Other

Uh oh!

kndehaan commented May 22, 2026

Uh oh!

louispt1 commented May 27, 2026

Uh oh!

noracato left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kndehaan commented Jun 1, 2026

Uh oh!

kndehaan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

louispt1 Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

noracato left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kndehaan left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

noracato commented Jun 11, 2026

Uh oh!

kndehaan commented Jun 11, 2026

Uh oh!

noracato commented Jun 11, 2026

Uh oh!

mabijkerk commented Jun 16, 2026

Uh oh!

kndehaan commented Jun 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

louispt1 commented May 21, 2026 •

edited by kndehaan

Loading

louispt1 Jun 2, 2026 •

edited

Loading

kndehaan left a comment •

edited

Loading