Missing properties from data-writer when generating data
Summary
When generating data with a '.generator' file, the resulting 'csv' is written without PropertiesSpecification in the top-most line of the 'csv' file.
Component
It happens when running the generator script, i.e. de.evoal.generator.main-plugin.
Steps to reproduce
Use the attached script 01-run-generator.sh with the attached data-description file for ackley and the attached generator file. Then open data/ackley.csv and check the header.
Current Behaviour
The current pipeline works fine, so the data is generated correctly, the ackley function is applied and values are written correctly as well. The PropertiesSpecification are (according to the logging information of EvoAl) correctly set right up until the application of the ackley function (de.evoal.languages.model.generator.impl.ApplyStatementImpl@1b5f960a (file: data/ackley.csv, count: 1000)). The log shows them to be
d.e.g.a.AbstractGeneratorFunction - Reading PropertiesSpecification [x:0, x:1]-
d.e.g.a.AbstractGeneratorFunction - Writing PropertiesSpecification [y:0].
Then, the pipeline switches to writing the 'csv' file (d.e.g.m.i.StatementExecutor - Writing data/ackley.csv with 1000 data points.). And at the subsequent call of final PropertiesSpecification specification = PropertiesSpecification.builder().build(); in class StatementExecutor, line 149 (give or take, I added some lines for debugging), the specification is empty (see the log: d.e.g.m.i.StatementExecutor - Properties after building following pipeline are PropertiesSpecification [], resulting from my addition of log.info("Properties after building following pipeline are {}", specification); in line 150.
This finally results in the following log-line: d.e.c.m.p.CSVPropertiesWriter - Creating CSV properties writer for PropertiesSpecification [] to file data/ackley.csv. which is consistent with what one gets, when running the current release.
Expected Behavior
I'd expect to see the PropertiesSpecifications of x:0, x:1, y:0 in the last log at least, as they would be printed to the first line of the csv file.
Relevant logs and/or screenshots
benchmark.ackley.ddl
module 'benchmark.ackley' {
types:
quotient type 'ackley-variable'
description: "Range information for input variables with Ackley-Benchmark-Function"
with constraints:
value < 5.12;
value > -5.12;
;
data:
/**
* Input dimension for the ackley generator function.
*/
real data 'x:0' of type 'ackley-variable';
real data 'x:1' of type 'ackley-variable';
/**
* Output dimension for the ackley generator function.
*/
quotient real data 'y:0';
}01-generate.ackley.generator
import "definitions" from de.evoal.core.math;
import "definitions" from de.evoal.generator.generator;
import "data" from benchmark.ackley;
module '01-generate.ackley' {
/**
* Introduce a simple pipeline that generates some test data using
* the ackley function.
*/
pipeline 'main-pipeline' [
/**
* First, we generate some normally distributed data.
*/
step {
component 'normal-distribution' {
'μ' := 0.0;
'σ' := 5.12;
}
writes [data 'x:0'];
}
step {
component 'normal-distribution' {
'μ' := 0.0;
'σ' := 5.12;
}
writes [data 'x:1'];
}
step {
component 'ackley' {
}
reads [data 'x:0', data 'x:1'];
writes [data 'y:0'];
}
]
write "data/ackley.csv" with 1000 samples from executing [ pipeline 'main-pipeline'];
}scripts/01-run-generator.sh
#!/bin/bash
export EVOAL_HOME=$( cd -- "$(dirname $0)/../evoal" >/dev/null 2>&1 ; pwd -P )
export EXECUTION_FOLDER=$( cd -- "$(dirname $0)/.." >/dev/null 2>&1 ; pwd -P )
export CONFIGURATION_FOLDER=$EXECUTION_FOLDER/configuration
$SHELL $EVOAL_HOME/bin/evoal-generator.sh $EXECUTION_FOLDER $CONFIGURATION_FOLDER/01-generate/ackley.generator -Bcore:search-path=$CONFIGURATION_FOLDERI replaced absolute paths with repo for privacy, but they are all correctly set. Main-Folder is supplementaryMaterial with the usual sub-folders configuration and scripts.
<-- loaded modules omitted -->
16:37:21.419 [main] INFO d.e.c.a.b.Blackboard - Setting blackboard entry from argument: '-Bcore:search-path=repo/2024-evoal-density/supplementaryMaterial/configuration'.
16:37:21.420 [main] INFO d.e.c.a.b.Blackboard - Binding entry core:search-path to repo/2024-evoal-density/supplementaryMaterial/configuration.
16:37:21.422 [main] INFO d.e.l.m.u.s.ClasspathGlobalScopeProvider - Adding repo/2024-evoal-density/supplementaryMaterial/configuration to EvoAl's search path.
16:37:21.422 [main] INFO d.e.c.a.b.Blackboard - Setting blackboard entry from argument: '-Bcore:logging=info'.
16:37:21.422 [main] INFO d.e.c.a.b.Blackboard - Binding entry core:logging to info.
16:37:21.422 [main] INFO d.e.c.a.b.Blackboard - Setting blackboard entry from argument: '-Bcore:main=data-generator'.
16:37:21.422 [main] INFO d.e.c.a.b.Blackboard - Binding entry core:main to data-generator.
16:37:21.422 [main] INFO d.e.c.a.b.Blackboard - Setting blackboard entry from argument: '-Bgenerator:configuration-file=repo/2024-evoal-density/supplementaryMaterial/configuration/01-generate/ackley.generator'.
16:37:21.422 [main] INFO d.e.c.a.b.Blackboard - Binding entry generator:configuration-file to repo/2024-evoal-density/supplementaryMaterial/configuration/01-generate/ackley.generator.
16:37:21.422 [main] INFO d.e.g.m.c.GeneratorModuleProducer - Using generator configuration from 'repo2024-evoal-density/supplementaryMaterial/configuration/01-generate/ackley.generator'.
16:37:22.251 [main] INFO d.e.g.m.c.GeneratorModuleProducer - Reading model file repo/2024-evoal-density/supplementaryMaterial/configuration/01-generate/ackley.generator.
16:37:22.421 [main] INFO d.e.l.m.u.s.ClasspathGlobalScopeProvider - Resolved file to classpath:/de/evoal/core/math.dl
16:37:22.422 [main] INFO d.e.l.m.u.s.ClasspathGlobalScopeProvider - Resolved file to classpath:/de/evoal/generator/generator.dl
16:37:22.422 [main] INFO d.e.l.m.u.s.ClasspathGlobalScopeProvider - Resolved file to repo/2024-evoal-density/supplementaryMaterial/configuration/benchmark/ackley.ddl
16:37:22.518 [main] INFO d.e.c.a.b.Blackboard - Binding entry generator:configuration to de.evoal.languages.model.generator.impl.GeneratorModuleImpl@15d0b458 (name: 01-generate.ackley).
16:37:22.518 [main] INFO d.e.c.m.Evoal - Fetching main class and handing over control
16:37:22.523 [main] INFO d.e.g.m.DataGenerator - Starting data generation.
16:37:22.524 [main] INFO d.e.g.m.DataGenerator - Creating pipeline 'main-pipeline'.
16:37:22.525 [main] INFO d.e.c.a.c.BeanFactory - Creating bean of type interface de.evoal.generator.api.GeneratorFunction with name de.evoal.generator.generator.normal-distribution.
16:37:22.529 [main] INFO d.e.g.a.AbstractGeneratorFunction - Reading PropertiesSpecification []
16:37:22.530 [main] INFO d.e.g.a.AbstractGeneratorFunction - Writing PropertiesSpecification [x:0]
16:37:22.531 [main] INFO d.e.g.m.g.NormalDistribution - Using distribution of μ=0.0 and σ=5.12
16:37:22.542 [main] INFO d.e.c.a.c.BeanFactory - Creating bean of type interface de.evoal.generator.api.GeneratorFunction with name de.evoal.generator.generator.normal-distribution.
16:37:22.542 [main] INFO d.e.g.a.AbstractGeneratorFunction - Reading PropertiesSpecification []
16:37:22.542 [main] INFO d.e.g.a.AbstractGeneratorFunction - Writing PropertiesSpecification [x:1]
16:37:22.542 [main] INFO d.e.g.m.g.NormalDistribution - Using distribution of μ=0.0 and σ=5.12
16:37:22.542 [main] INFO d.e.c.a.c.BeanFactory - Creating bean of type interface de.evoal.generator.api.GeneratorFunction with name de.evoal.generator.generator.ackley.
16:37:22.543 [main] INFO d.e.g.a.AbstractGeneratorFunction - Reading PropertiesSpecification [x:0, x:1]
16:37:22.543 [main] INFO d.e.g.a.AbstractGeneratorFunction - Writing PropertiesSpecification [y:0]
16:37:22.543 [main] INFO d.e.c.a.l.ExpressionEvaluator - Attribute binding not found. Using default value.
16:37:22.543 [main] INFO d.e.c.a.l.ExpressionEvaluator - Attribute binding not found. Using default value.
16:37:22.543 [main] INFO d.e.c.a.l.ExpressionEvaluator - Attribute binding not found. Using default value.
16:37:22.546 [main] INFO d.e.g.m.b.Ackley - Creating Ackley function with parameter a=20.0, b=0.2 and c=6.283185307.
16:37:22.546 [main] INFO d.e.g.m.DataGenerator - Statements of Pipeline are [de.evoal.languages.model.generator.impl.ApplyStatementImpl@323b0632 (file: data/ackley.csv, count: 1000)] // added by my debugging, not in normal output
16:37:22.549 [main] INFO d.e.g.m.i.StatementExecutor - Writing data/ackley.csv with 1000 data points.
16:37:22.549 [main] INFO d.e.g.m.i.StatementExecutor - Properties after building following pipeline are PropertiesSpecification [] // added by my debugging, not in normal output
16:37:22.550 [main] INFO d.e.g.m.i.StatementExecutor - Properties after building following pipeline are PropertiesSpecification [] // added by my debugging, not in normal output
16:37:22.551 [main] INFO d.e.c.a.p.i.PropertiesIOFactory - Specification just before creating writer is PropertiesSpecification [] // added by my debugging, not in normal output
16:37:22.551 [main] INFO d.e.c.a.c.BeanFactory - Creating bean of type interface de.evoal.core.api.properties.io.PropertiesWriter with name csv-writer.
16:37:22.551 [main] INFO d.e.c.m.p.CSVPropertiesWriter - Creating CSV properties writer for PropertiesSpecification [] to file data/ackley.csv.
16:37:22.599 [main] INFO d.e.g.m.DataGenerator - Finished data generation.
16:37:22.599 [main] INFO d.e.c.m.Evoal - Shutting down CDI container
16:37:22.673 [main] INFO o.j.w.Bootstrap - WELD-ENV-002001: Weld SE container 4ba77111-73bf-4bdd-9e98-6fa0a6d8bca4 shut downPossible fixes
I think it happens somewhere in the applyStatement of the StatementExecutor. Generating the data still works, but after applying the function, the PropertiesSpecification is empty.
Contact Person
/cc @berber