Skip to content

Commit 4ddc3a9

Browse files
xaviermarchegayxaviermarchegay
committed
feat doc
1 parent 2be4cd5 commit 4ddc3a9

54 files changed

Lines changed: 2232 additions & 50 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/cookbooks/01-common_setup.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,3 +45,54 @@ monolog:
4545
max_files: 10
4646
channels: ['cleverage_process_task', 'cleverage_process_transformer']
4747
```
48+
49+
Example: lightweight file import
50+
--------------------------------
51+
52+
To give more context on how configuration ties to the PHP code, here is a minimal file-import workflow built with classes already shipped in `src/Task/File` and `src/Task/Reporting`.
53+
54+
```yaml
55+
clever_age_process:
56+
configurations:
57+
app.file_import:
58+
default_error_strategy: stop
59+
tasks:
60+
read_csv:
61+
service: '@CleverAge\ProcessBundle\Task\File\InputFileReaderTask'
62+
options:
63+
file_path: '%kernel.project_dir%/data/products.csv'
64+
format: csv
65+
outputs: [split_rows]
66+
67+
split_rows:
68+
service: '@CleverAge\ProcessBundle\Task/File/Csv/CsvSplitterTask'
69+
options:
70+
delimiter: ';'
71+
outputs: [transform]
72+
73+
transform:
74+
service: '@CleverAge\ProcessBundle\Task\TransformerTask'
75+
options:
76+
transformers:
77+
mapping:
78+
mapping:
79+
id: { code: '[id]' }
80+
slug:
81+
code:
82+
- '[name]'
83+
- '[category]'
84+
transformers:
85+
implode:
86+
separator: '-'
87+
outputs: [write_csv]
88+
89+
write_csv:
90+
service: '@CleverAge\ProcessBundle\Task\File/Csv/CsvWriterTask'
91+
options:
92+
file_path: '%kernel.project_dir%/var/output/products_prepared.csv'
93+
headers:
94+
- id
95+
- slug
96+
```
97+
98+
Each task above maps to the concrete classes such as `InputFileReaderTask`, `CsvSplitterTask` and `CsvWriterTask` found under `src/Task/File`. Mentioning the `default_error_strategy` helps downstream code in `src/Configuration/ProcessConfiguration.php` know how to propagate failures.

‎docs/cookbooks/etl_aggregate_reports.md‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
ETL report aggregation
2+
======================
3+
4+
This example shows an ETL path that reads several JSON log files, aggregates rates per service, and writes a CSV summary that can feed a dashboard.
5+
6+
```yaml
7+
clever_age_process:
8+
configurations:
9+
app.etl_report_aggregate:
10+
default_error_strategy: stop
11+
tasks:
12+
list_sources:
13+
service: '@CleverAge\ProcessBundle\Task\File\FolderBrowserTask'
14+
options:
15+
folder: '%kernel.project_dir%/data/logs'
16+
filter: '*.json'
17+
outputs: [read_log]
18+
19+
read_log:
20+
service: '@CleverAge\ProcessBundle\Task\File\JsonStream\JsonStreamReaderTask'
21+
options:
22+
file_path: '[file]' # value injected from FolderBrowserTask
23+
iterator: items
24+
outputs: [map_log]
25+
26+
map_log:
27+
service: '@CleverAge\ProcessBundle\Task\TransformerTask'
28+
options:
29+
transformers:
30+
mapping:
31+
mapping:
32+
service: { code: '[service]' }
33+
duration: { code: '[duration]' }
34+
status: { code: '[status]' }
35+
outputs: [group_reports]
36+
37+
group_reports:
38+
service: '@CleverAge\ProcessBundle\Task\GroupByAggregateIterableTask'
39+
options:
40+
group_by: service
41+
aggregate:
42+
duration: { type: 'avg' }
43+
outputs: [write_summary]
44+
45+
write_summary:
46+
service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvWriterTask'
47+
options:
48+
file_path: '%kernel.project_dir%/var/exports/report_summary.csv'
49+
headers: [service, duration]
50+
```
51+
52+
The `FolderBrowserTask`, `JsonStreamReaderTask`, `TransformerTask`, and `GroupByAggregateIterableTask` classes live under `src/Task/File` and `src/Task`. They demonstrate a reusable read/transform/aggregate path for any automation workflow.

‎docs/cookbooks/etl_file_sync.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
File synchronization ETL
2+
=======================
3+
4+
This recipe describes a typical ETL flow: read a CSV file, enrich the data with transformations, then write to another file while keeping a statistics log.
5+
6+
```yaml
7+
clever_age_process:
8+
configurations:
9+
app.etl_file_sync:
10+
default_error_strategy: stop
11+
tasks:
12+
load_source:
13+
service: '@CleverAge\ProcessBundle\Task\File\InputFileReaderTask'
14+
options:
15+
file_path: '%kernel.project_dir%/data/catalog.csv'
16+
format: csv
17+
outputs: [split_rows]
18+
19+
split_rows:
20+
service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvSplitterTask'
21+
options:
22+
delimiter: ';'
23+
outputs: [normalize]
24+
25+
normalize:
26+
service: '@CleverAge\ProcessBundle\Task\TransformerTask'
27+
options:
28+
transformers:
29+
mapping:
30+
mapping:
31+
sku: { code: '[sku]' }
32+
price: { code: '[price]', transformers: { cast: { type: float } } }
33+
sent_at:
34+
code: 'format_datetime("[updated_at]", "Y-m-d")'
35+
outputs: [deduplicate]
36+
37+
deduplicate:
38+
service: '@CleverAge\ProcessBundle\Task\FilterTask'
39+
options:
40+
unique_field: sku
41+
outputs: [write_target]
42+
43+
write_target:
44+
service: '@CleverAge\ProcessBundle\Task\File\Csv\CsvWriterTask'
45+
options:
46+
file_path: '%kernel.project_dir%/var/exports/catalog_normalized.csv'
47+
headers: [sku,price,sent_at]
48+
outputs: [log_stats]
49+
50+
log_stats:
51+
service: '@CleverAge\ProcessBundle\Task\Reporting\StatCounterTask'
52+
options:
53+
increment:
54+
rows: 1
55+
```
56+
57+
The `FilterTask` in the sequence corresponds to `src/Task/FilterTask.php`, and the CSV tasks come from `src/Task/File/Csv`. The final counter uses `src/Task/Reporting/StatCounterTask.php` to show how to append a simple metric to the logs.

‎docs/cookbooks/memory_usage_graph.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,3 +38,8 @@ To graph the output of this process, use this Gnuplot command in your host envir
3838
```bash
3939
$ gnuplot -e 'while(1) {plot "memory.dat" using 0:1 with lines; pause 1; reread}'
4040
```
41+
42+
Alternative: log memory per process phase
43+
-------------------------------------------
44+
45+
If you want more granular analysis, add multiple `TransformerTask` steps that call `memory_get_usage` before and after a critical moment in your process (e.g., before and after the `TransformerTask` in the CSV import flow above). Dump each value to `memory_phase.dat` with a `CsvWriterTask` that adds `phase` and `memory_usage` columns so you can compare how much heap each phase consumes.

‎docs/cookbooks/performances_monitoring.md‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,4 +46,15 @@ CPU Time n/a
4646
Memory 5.35MB
4747
Network n/a n/a n/a
4848
SQL n/a n/a
49-
```
49+
```
50+
51+
Example: measuring CSV processing throughput
52+
------------------------------------------------
53+
54+
When you run `cleverage:process:execute` you already benefit from the `ProcessLogger` that records task timing stats under `src/Logger`. To highlight actionable Blackfire runs, log the following steps:
55+
56+
```bash
57+
blackfire run php bin/console --env=prod cleverage:process:execute app.file_import
58+
```
59+
60+
After the JSON output, look for the `ProcessLogger` section (e.g., `timing.cleverage_process.task`) and correlate it with the CSV reader/writer tasks you configured in `docs/cookbooks/01-common_setup.md`. Use those duration metrics to decide whether to parallelize `CsvSplitterTask` with `TransformerTask`, or to revisit buffering logic in `CsvWriterTask`.

‎docs/index.md‎

Lines changed: 51 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@
66
- [Advanced workflow](04-advanced_workflow.md)
77
- Cookbooks
88
- [Common Setup](cookbooks/01-common_setup.md)
9+
- [ETL file synchronization](cookbooks/etl_file_sync.md)
10+
- [ETL report aggregation](cookbooks/etl_aggregate_reports.md)
911
- [Transformations]
1012
- [Flow manipulation]
1113
- [Dummy tasks]
@@ -32,18 +34,18 @@
3234
- Data manipulation and transformations
3335
- [DenormalizerTask](reference/tasks/denormalizer_task.md)
3436
- [NormalizerTask](reference/tasks/normalizer_task.md)
35-
- [DeserializerTask]
36-
- [SerializerTask]
37+
- [DeserializerTask](reference/tasks/deserializer_task.md)
38+
- [SerializerTask](reference/tasks/serializer_task.md)
3739
- [PropertyGetterTask](reference/tasks/property_getter_task.md)
3840
- [PropertySetterTask](reference/tasks/property_setter_task.md)
39-
- [ObjectUpdaterTask]
40-
- [SplitJoinLineTask]
41+
- [ObjectUpdaterTask](reference/tasks/object_updater_task.md)
42+
- [SplitJoinLineTask](reference/tasks/split_join_line_task.md)
4143
- [TransformerTask](reference/tasks/transformer_task.md)
42-
- [ValidatorTask]
44+
- [ValidatorTask](reference/tasks/validator_task.md)
4345
- File/CSV
4446
- [CsvReaderTask](reference/tasks/csv_reader_task.md)
4547
- [CsvWriterTask](reference/tasks/csv_writer_task.md)
46-
- [CSVSplitterTask]
48+
- [CsvSplitterTask](reference/tasks/csv_splitter_task.md)
4749
- [InputCsvReaderTask](reference/tasks/input_csv_reader_task.md)
4850
- File/JsonStream
4951
- [JsonStreamReaderTask](reference/tasks/json_stream_reader_task.md)
@@ -52,14 +54,14 @@
5254
- [XmlReaderTask](reference/tasks/xml_reader_task.md)
5355
- [XmlWriterTask](reference/tasks/xml_writer_task.md)
5456
- File/Yaml
55-
- [YamlReaderTask]
56-
- [YamlWriterTask]
57+
- [YamlReaderTask](reference/tasks/yaml_reader_task.md)
58+
- [YamlWriterTask](reference/tasks/yaml_writer_task.md)
5759
- File
58-
- [FileMoverTask]
60+
- [FileMoverTask](reference/tasks/file_mover_task.md)
5961
- [FileReaderTask](reference/tasks/file_reader_task.md)
60-
- [FileRemoverTask]
62+
- [FileRemoverTask](reference/tasks/file_remover_task.md)
6163
- [FileSplitterTask](reference/tasks/file_splitter_task.md)
62-
- [FileWriterTask]
64+
- [FileWriterTask](reference/tasks/file_writer_task.md)
6365
- [FolderBrowserTask](reference/tasks/folder_browser_task.md)
6466
- [InputFileReaderTask](reference/tasks/input_file_reader_task.md)
6567
- [InputFolderBrowserTask](reference/tasks/input_folder_browser_task.md)
@@ -69,62 +71,62 @@
6971
- [AggregateIterableTask](reference/tasks/aggregate_iterable_task.md)
7072
- [InputAggregatorTask](reference/tasks/input_aggregator_task.md)
7173
- [InputIteratorTask](reference/tasks/input_iterator_task.md)
72-
- [ArrayMergeTask]
73-
- [ColumnAggregatorTask]
74-
- [RowAggregatorTask]
75-
- [FilterTask]
76-
- [GroupByAggregateIterableTask]
77-
- [SimpleBatchTask]
78-
- [IterableBatchTask]
79-
- [SkipEmptyTask]
80-
- [StopTask]
74+
- [ArrayMergeTask](reference/tasks/array_merge_task.md)
75+
- [ColumnAggregatorTask](reference/tasks/column_aggregator_task.md)
76+
- [RowAggregatorTask](reference/tasks/row_aggregator_task.md)
77+
- [FilterTask](reference/tasks/filter_task.md)
78+
- [GroupByAggregateIterableTask](reference/tasks/group_by_aggregate_iterable_task.md)
79+
- [SimpleBatchTask](reference/tasks/simple_batch_task.md)
80+
- [IterableBatchTask](reference/tasks/iterable_batch_task.md)
81+
- [SkipEmptyTask](reference/tasks/skip_empty_task.md)
82+
- [StopTask](reference/tasks/stop_task.md)
8183
- Process
82-
- [CommandRunnerTask]
83-
- [ProcessExecutorTask]
84-
- [ProcessLauncherTask]
84+
- [CommandRunnerTask](reference/tasks/command_runner_task.md)
85+
- [ProcessExecutorTask](reference/tasks/process_executor_task.md)
86+
- [ProcessLauncherTask](reference/tasks/process_launcher_task.md)
8587
- Reporting
86-
- [AdvancedStatCounterTask]
88+
- [AdvancedStatCounterTask](reference/tasks/advanced_stat_counter_task.md)
8789
- [LoggerTask](reference/tasks/logger_task.md)
88-
- [StatCounterTask]
90+
- [StatCounterTask](reference/tasks/stat_counter_task.md)
8991
- Transformers
9092
- Basic and debug
91-
- [CachedTransformer]
92-
- [CallbackTransformer]
93-
- [CastTransformer]
94-
- [ConstantTransformer]
95-
- [ConvertValueTransformer]
96-
- [DebugTransformer]
97-
- [DefaultTransformer]
93+
- [CachedTransformer](reference/transformers/cached_transformer.md)
94+
- [CallbackTransformer](reference/transformers/callback_transformer.md)
95+
- [CastTransformer](reference/transformers/cast_transformer.md)
96+
- [ConstantTransformer](reference/transformers/constant_transformer.md)
97+
- [ConvertValueTransformer](reference/transformers/convert_value_transformer.md)
98+
- [DebugTransformer](reference/transformers/debug_transformer.md)
99+
- [DefaultTransformer](reference/transformers/default_transformer.md)
98100
- [GenericTransformer]
99-
- [EvaluatorTransformer]
100-
- [ExpressionLanguageMapTransformer]
101+
- [EvaluatorTransformer](reference/transformers/evaluator_transformer.md)
102+
- [ExpressionLanguageMapTransformer](reference/transformers/expression_language_map_transformer.md)
101103
- [MappingTransformer](reference/transformers/mapping_transformer.md)
102104
- [MultiReplaceTransformer](reference/transformers/multi_replace_transformer.md)
103-
- [PregFilterTransformer]
105+
- [PregFilterTransformer](reference/transformers/preg_filter_transformer.md)
104106
- [RulesTransformer](reference/transformers/rules_transformer.md)
105-
- [TypeSetterTransformer]
106-
- [UnsetTransformer]
107-
- [WrapperTransformer]
107+
- [TypeSetterTransformer](reference/transformers/type_setter_transformer.md)
108+
- [UnsetTransformer](reference/transformers/unset_transformer.md)
109+
- [WrapperTransformer](reference/transformers/wrapper_transformer.md)
108110
- Array
109-
- [ArrayElementTransformer]
111+
- [ArrayElementTransformer](reference/transformers/array_element_transformer.md)
110112
- [ArrayFilterTransformer](reference/transformers/array_filter_transformer.md)
111-
- [ArrayFirstTransformer]
112-
- [ArrayLastTransformer]
113+
- [ArrayFirstTransformer](reference/transformers/array_first_transformer.md)
114+
- [ArrayLastTransformer](reference/transformers/array_last_transformer.md)
113115
- [ArrayMapTransformer](reference/transformers/array_map_transformer.md)
114-
- [ArrayUnsetTransformer]
116+
- [ArrayUnsetTransformer](reference/transformers/array_unset_transformer.md)
115117
- Date
116118
- [DateFormatTransformer](reference/transformers/date_format.md)
117119
- [DateParserTransformer](reference/transformers/date_parser.md)
118120
- Object
119-
- [InstantiateTransformer]
120-
- [PropertyAccessorTransformer]
121-
- [RecursivePropertySetterTransformer]
121+
- [InstantiateTransformer](reference/transformers/instantiate_transformer.md)
122+
- [PropertyAccessorTransformer](reference/transformers/property_accessor_transformer.md)
123+
- [RecursivePropertySetterTransformer](reference/transformers/recursive_property_setter_transformer.md)
122124
- Serialization
123-
- [DenormalizeTransformer]
124-
- [NormalizeTransformer]
125+
- [DenormalizeTransformer](reference/transformers/denormalize_transformer.md)
126+
- [NormalizeTransformer](reference/transformers/normalize_transformer.md)
125127
- String
126-
- [ExplodeTransformer]
127-
- [HashTransformer]
128+
- [ExplodeTransformer](reference/transformers/explode_transformer.md)
129+
- [HashTransformer](reference/transformers/hash_transformer.md)
128130
- [ImplodeTransformer](reference/transformers/implode_transformer.md)
129131
- [PregMatchTransformer](reference/transformers/preg_match_transformer.md)
130132
- [SlugifyTransformer](reference/transformers/slugify_transformer.md)

‎docs/reference/tasks/advanced_stat_counter_task.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
AdvancedStatCounterTask
2+
=======================
3+
4+
Log performance statistics at regular intervals during iteration. Displays time between iterations, items per
5+
second rate, total items processed, and total elapsed time.
6+
7+
Task reference
8+
--------------
9+
10+
* **Service**: `CleverAge\ProcessBundle\Task\Reporting\AdvancedStatCounterTask`
11+
12+
Accepted inputs
13+
---------------
14+
15+
`any`: input is not used, only the iteration count matters
16+
17+
Possible outputs
18+
----------------
19+
20+
No output is set (the task skips most iterations). Statistics are logged via the logger.
21+
22+
Options
23+
-------
24+
25+
| Code | Type | Required | Default | Description |
26+
|--------------|-----------|:--------:|---------|-------------------------------------------------------------------|
27+
| `num_items` | `integer` | | `1` | Number of logical items per iteration (multiplier for rate calc) |
28+
| `skip_first` | `integer` | | `0` | Number of initial iterations to skip before tracking |
29+
| `show_every` | `integer` | | `1` | Display statistics every N iterations |
30+
31+
Example
32+
-------
33+
34+
```yaml
35+
# Task configuration level
36+
stats:
37+
service: '@CleverAge\ProcessBundle\Task\Reporting\AdvancedStatCounterTask'
38+
options:
39+
show_every: 100
40+
num_items: 1
41+
```
42+
43+
Output in logs (every 100 iterations):
44+
```
45+
Last iteration 00:00:12 ago - 1,50 items/s - 500 items processed in 00:05:33
46+
```

0 commit comments

Comments
 (0)