From cbc20f27ffc40503e68169526e3e0db80ab7d4af Mon Sep 17 00:00:00 2001 From: Dave Mills Date: Fri, 31 Jul 2026 12:27:28 +0100 Subject: [PATCH 1/4] Remove legacy Farm model, FarmResource, and HOLPA survey-data models Completes Phase 6 (cutover) of the ODK-Entities farm CRUD work: farms now live only as ODK Central entities via FarmEntity/FarmEntityResource. - move FarmListHeaderWidget and the combined-import Blade view into the FarmEntityResource tree (both were shared, not legacy) - delete FarmResource + pages, App\Models\SampleFrame\Farm, FarmImport, FarmPolicy, FarmImportCompleted, DataCollectionByFarm and HelperService::findFarmLocationDetails() - all dormant or unreferenced - purge the whole App\Models\SurveyData namespace (FarmSurveyData, Crop, Livestock) and the RepeatModel interface that only described it; the same-named exports survive, they are Dataset-name-driven not model-driven - repoint Team::pilotProgress / dataCollectionProgress at submissions (the farm-derived expressions could only ever return false) - drop Location's three farm-completion attributes and the dead 'Farms surveyed' admin stat; keep farms_all_count (FarmEntity-backed) - ImportFarmsAction now tags its Import record model_type => FarmEntity - drop the farms, crops, livestocks and farm_survey_data tables --- CLAUDE.md | 3 +- app/Events/FarmImportCompleted.php | 36 -- app/Exports/DataExport/DatasetExport.php | 10 +- .../Admin/Widgets/DataCollectedWidget.php | 9 +- .../Resources/FarmEntityResource.php | 10 +- .../Pages/ImportLocationsAndFarmEntities.php | 11 +- .../Pages/ListFarmEntities.php | 10 +- .../Widgets/FarmListHeaderWidget.php | 6 +- .../LocationLevels/Resources/FarmResource.php | 183 ---------- .../Pages/ImportLocationsAndFarms.php | 312 ------------------ .../FarmResource/Pages/ListFarms.php | 76 ----- .../Resources/LocationLevelResource.php | 3 - .../Tables/Actions/ImportFarmsAction.php | 4 +- app/Imports/FarmEntityImport.php | 5 +- app/Imports/FarmImport.php | 191 ----------- .../DataCollection/DataCollectionByFarm.php | 67 ---- .../DataCollectionByLocation.php | 35 +- app/Models/Interfaces/RepeatModel.php | 17 - app/Models/SampleFrame/Farm.php | 110 ------ app/Models/SampleFrame/Location.php | 47 --- app/Models/SurveyData/Crop.php | 27 -- app/Models/SurveyData/FarmSurveyData.php | 43 --- app/Models/SurveyData/Livestock.php | 28 -- app/Models/Team.php | 16 +- app/Policies/FarmEntityPolicy.php | 2 - app/Policies/FarmPolicy.php | 49 --- app/Providers/AppServiceProvider.php | 3 - app/Services/HelperService.php | 34 +- app/Services/OdkFarmEntityService.php | 6 +- ...024_06_11_06_104524_create_farms_table.php | 50 --- ...1_121716_create_farm_survey_data_table.php | 59 ---- ...024_08_12_12_111301_create_crops_table.php | 134 -------- ...8_12_12_111310_create_livestocks_table.php | 43 --- ...6_07_31_000001_drop_legacy_farm_tables.php | 19 ++ database/seeders/Prep/DatasetSeeder.php | 15 +- ...ort-locations-and-farm-entities.blade.php} | 0 .../survey-locations-index.blade.php | 3 - .../widgets/farm-list-header-widget.blade.php | 0 .../data-collection-by-farm.blade.php | 5 - tests/Feature/Crud/AppPanelCrudTest.php | 36 +- .../Exports/FarmSurveyDataExportTest.php | 28 ++ tests/Feature/Smoke/AppPanelTest.php | 4 +- 42 files changed, 123 insertions(+), 1626 deletions(-) delete mode 100644 app/Events/FarmImportCompleted.php rename app/Filament/App/Clusters/LocationLevels/Resources/{FarmResource => FarmEntityResource}/Widgets/FarmListHeaderWidget.php (80%) delete mode 100644 app/Filament/App/Clusters/LocationLevels/Resources/FarmResource.php delete mode 100644 app/Filament/App/Clusters/LocationLevels/Resources/FarmResource/Pages/ImportLocationsAndFarms.php delete mode 100644 app/Filament/App/Clusters/LocationLevels/Resources/FarmResource/Pages/ListFarms.php delete mode 100644 app/Imports/FarmImport.php delete mode 100644 app/Livewire/DataCollection/DataCollectionByFarm.php delete mode 100644 app/Models/Interfaces/RepeatModel.php delete mode 100644 app/Models/SampleFrame/Farm.php delete mode 100644 app/Models/SurveyData/Crop.php delete mode 100644 app/Models/SurveyData/FarmSurveyData.php delete mode 100644 app/Models/SurveyData/Livestock.php delete mode 100644 app/Policies/FarmPolicy.php delete mode 100644 database/migrations/06_locations/2024_06_11_06_104524_create_farms_table.php delete mode 100644 database/migrations/07_survey_data/2024_08_12_11_121716_create_farm_survey_data_table.php delete mode 100644 database/migrations/07_survey_data/2024_08_12_12_111301_create_crops_table.php delete mode 100644 database/migrations/07_survey_data/2024_08_12_12_111310_create_livestocks_table.php create mode 100644 database/migrations/2026_07_31_000001_drop_legacy_farm_tables.php rename resources/views/filament/app/clusters/location-levels/resources/{farm-resource/pages/import-locations-and-farms.blade.php => farm-entity-resource/pages/import-locations-and-farm-entities.blade.php} (100%) rename resources/views/filament/app/resources/{farm-resource => farm-entity-resource}/widgets/farm-list-header-widget.blade.php (100%) delete mode 100644 resources/views/livewire/data-collection/data-collection-by-farm.blade.php create mode 100644 tests/Feature/Exports/FarmSurveyDataExportTest.php diff --git a/CLAUDE.md b/CLAUDE.md index 7c230fda..0303fa48 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -67,8 +67,7 @@ Models in this app often extend base classes from these packages (e.g., `User`, ### Model Domain Structure Models are namespaced by domain under `app/Models/`: -- `Models/SampleFrame/` — Farm, Location, LocationLevel (the survey sample population) -- `Models/SurveyData/` — FarmSurveyData, Crop, Product and other ODK submission data +- `Models/SampleFrame/` — FarmEntity, Location, LocationLevel (the survey sample population) - `Models/Holpa/` — LocalIndicator, Theme, Domain (custom indicator framework) - `Models/Reference/` — Reference/lookup data diff --git a/app/Events/FarmImportCompleted.php b/app/Events/FarmImportCompleted.php deleted file mode 100644 index 9f5befd7..00000000 --- a/app/Events/FarmImportCompleted.php +++ /dev/null @@ -1,36 +0,0 @@ - - */ - public function broadcastOn(): array - { - return [ - new Channel('xlsforms'), - ]; - } -} diff --git a/app/Exports/DataExport/DatasetExport.php b/app/Exports/DataExport/DatasetExport.php index e87f974d..c995b9c6 100644 --- a/app/Exports/DataExport/DatasetExport.php +++ b/app/Exports/DataExport/DatasetExport.php @@ -1,7 +1,7 @@ get(); } - /** - * @return Collection - */ public function collection(): Collection { return $this->entities->map(function (Entity $entity) { // get the farm_id and farm_name from the owner relationship - /** @var Farm $farm */ $farm = $entity->submission->primaryDataSubject; $row = [ @@ -51,6 +46,7 @@ public function collection(): Collection $value = $entity->values->firstWhere('dataset_variable_name', $heading); $row[$heading] = $value ? $value->value : null; } + return $row; }); } diff --git a/app/Filament/Admin/Widgets/DataCollectedWidget.php b/app/Filament/Admin/Widgets/DataCollectedWidget.php index cecb40ee..4487014a 100644 --- a/app/Filament/Admin/Widgets/DataCollectedWidget.php +++ b/app/Filament/Admin/Widgets/DataCollectedWidget.php @@ -2,7 +2,6 @@ namespace App\Filament\Admin\Widgets; -use App\Models\SampleFrame\Farm; use Filament\Widgets\StatsOverviewWidget; use Filament\Widgets\StatsOverviewWidget\Stat; use Illuminate\Support\HtmlString; @@ -16,14 +15,8 @@ protected function getStats(): array { $result = []; - // find number of farms that completed both household form and fieldwork form - $farmsSurveyed = Farm::where('household_form_completed', true)->where('fieldwork_form_completed', true)->count(); - // $farmsSurveyed = Role::count(); - - array_push($result, Stat::make(new HtmlString('Farms surveyed'), $farmsSurveyed)); - // find total number of submissions for each xlsform template - $xlsformTemplates = XlsFormTemplate::all(); + $xlsformTemplates = XlsformTemplate::all(); foreach ($xlsformTemplates as $xlsformTemplate) { $total = 0; diff --git a/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource.php b/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource.php index 3dbd9db0..87b7dfaa 100644 --- a/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource.php +++ b/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource.php @@ -26,9 +26,8 @@ use Filament\Tables\Table; use Illuminate\Validation\Rules\Unique; -// New, ODK-Central-Entities-backed Farm CRUD page, built alongside the existing -// FarmResource (not replacing it yet). See docs/plans/odk-entities-farm-crud.md. -// Nav-hidden, same as FarmResource - reachable by direct link during development/testing. +// ODK-Central-Entities-backed Farm CRUD page. See docs/plans/odk-entities-farm-crud.md. +// Nav-hidden - reached via the Survey Locations index and the location-levels cluster nav. class FarmEntityResource extends Resource { protected static ?string $model = FarmEntity::class; @@ -108,13 +107,12 @@ public static function table(Table $table): Table $dataset = $service->ensureDataset($team); // Dynamic identifier/property columns are driven by DatasetVariable (schema-level, - // stays local), not by scanning records' JSON keys like the old FarmResource does. - // Values themselves are never persisted locally - $livewire->liveFarmData is the + // stays local). Values themselves are never persisted locally - $livewire->liveFarmData is the // live feed ListFarmEntities::mount() fetched for this page load (see // OdkFarmEntityService::refreshFromCentral()). The location cascade attributes // (loc{n}/loc{n}_name/loc{n}_type) and GPS are excluded here via the 'loc' // classification - GPS has its own dedicated form fields and the cascade attributes - // are owned by the dedicated Location column, matching the old FarmResource. + // are owned by the dedicated Location column. $propertyColumns = $dataset->variables() ->where('name', '!=', 'team_code') ->get() diff --git a/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource/Pages/ImportLocationsAndFarmEntities.php b/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource/Pages/ImportLocationsAndFarmEntities.php index 1e677493..47674362 100644 --- a/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource/Pages/ImportLocationsAndFarmEntities.php +++ b/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource/Pages/ImportLocationsAndFarmEntities.php @@ -30,18 +30,15 @@ use Maatwebsite\Excel\Facades\Excel; use Maatwebsite\Excel\HeadingRowImport; -// ODK-Entities-backed counterpart to FarmResource\Pages\ImportLocationsAndFarms - the -// column-mapping wizard is identical (it's about parsing a spreadsheet + location -// hierarchy, independent of storage backend). Only the farm half of save() differs: -// FarmEntityImport instead of FarmImport. Locations stay fully local either way. Reuses -// the same Blade view as the original page - it's generic form+actions boilerplate. +// Combined wizard: one spreadsheet holding both the location hierarchy and the farm list. +// Locations are imported into local tables; farms are pushed to ODK Central as entities. class ImportLocationsAndFarmEntities extends Page implements HasForms { use InteractsWithForms; protected static string $resource = FarmEntityResource::class; - protected string $view = 'filament.app.clusters.location-levels.resources.farm-resource.pages.import-locations-and-farms'; + protected string $view = 'filament.app.clusters.location-levels.resources.farm-entity-resource.pages.import-locations-and-farm-entities'; public function getTitle(): string { @@ -79,7 +76,7 @@ public function save(): void // copy it as a duplicate, which will be stored with the import model for farms Storage::copy($data['upload'], $data['upload'].'_duplicate'); - // No "replace all locations" option here (unlike the old FarmResource wizard) - + // No "replace all locations" option here - // farm_entities.location_id used to cascadeOnDelete, which combined with mass // location deletion here into a real bug (see docs/plans/odk-entities-farm-crud.md). // The FK is now nullOnDelete instead, but a bulk "delete every location" action is diff --git a/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource/Pages/ListFarmEntities.php b/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource/Pages/ListFarmEntities.php index b7989b24..986a8bca 100644 --- a/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource/Pages/ListFarmEntities.php +++ b/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource/Pages/ListFarmEntities.php @@ -3,7 +3,7 @@ namespace App\Filament\App\Clusters\LocationLevels\Resources\FarmEntityResource\Pages; use App\Filament\App\Clusters\LocationLevels\Resources\FarmEntityResource; -use App\Filament\App\Clusters\LocationLevels\Resources\FarmResource\Widgets\FarmListHeaderWidget; +use App\Filament\App\Clusters\LocationLevels\Resources\FarmEntityResource\Widgets\FarmListHeaderWidget; use App\Filament\App\Pages\SurveyDashboard; use App\Filament\App\Pages\SurveyLocations\SurveyLocationsIndex; use App\Filament\Tables\Actions\ImportFarmsAction; @@ -34,8 +34,6 @@ public function getBreadcrumbs(): array protected function getHeaderWidgets(): array { return [ - // Reused as-is from FarmResource - it's a generic instructions panel with no - // Farm-model-specific logic. FarmListHeaderWidget::class, ]; } @@ -58,8 +56,6 @@ public function mount(): void protected function getHeaderActions(): array { return [ - // Divert to the combined wizard - identical to ListFarms' equivalent button, - // just pointed at this resource's own import route. Action::make('import') ->label(fn () => t('Import Locations and Farm List')) ->extraAttributes(['class' => 'buttonb']) @@ -70,9 +66,7 @@ protected function getHeaderActions(): array // Reuses the existing column-mapping modal as-is (it's about parsing a // spreadsheet, independent of storage backend) - only the underlying import - // class differs. NOTE: the Import audit record this creates is tagged - // model_type => Farm::class regardless (hardcoded in ImportFarmsAction), which - // is cosmetically inaccurate for entity imports but doesn't affect behaviour. + // class differs. ImportFarmsAction::make() ->color('primary') ->extraAttributes(['class' => 'buttonb']) diff --git a/app/Filament/App/Clusters/LocationLevels/Resources/FarmResource/Widgets/FarmListHeaderWidget.php b/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource/Widgets/FarmListHeaderWidget.php similarity index 80% rename from app/Filament/App/Clusters/LocationLevels/Resources/FarmResource/Widgets/FarmListHeaderWidget.php rename to app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource/Widgets/FarmListHeaderWidget.php index b5631b00..71943353 100644 --- a/app/Filament/App/Clusters/LocationLevels/Resources/FarmResource/Widgets/FarmListHeaderWidget.php +++ b/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource/Widgets/FarmListHeaderWidget.php @@ -1,13 +1,13 @@ user()->latestTeam->locationLevels->where('has_farms', 1)->first(); - - return $schema - ->components([ - - Hidden::make('owner_id') - ->default(HelperService::getCurrentOwner()->id), - Select::make('location_id') - ->label(t('Select the') . ' ' . $locationLevelWithFarms->name . ' ' . t('for this farm')) - ->options($locationLevelWithFarms->locations->pluck('name', 'id')), - - TextInput::make('team_code') - ->label(t('Unique code')) - ->helperText(t('Please enter a unique code to identify this farm for your team')) - // team code should be unique per team, as other teams may have the same team code - ->unique(modifyRuleUsing: function (Unique $rule) { - return $rule->where('owner_id', HelperService::getCurrentOwner()->id); - }) - ->maxLength(255), - - Section::make(t('Personally Identifiable information')) - ->description(t('This section lets you add any information about the farm or farmer that lets your enumerators personally identify the farm / farmer.')) - ->schema([ - KeyValue::make('identifiers') - ->hint(t('For example: farm name, name of household head, phone number, physical address.')) - ->helperText(t('Information added here will be available to your team through data downloads, and if required can be included in the ODK survey to help enumerators ensure they reach the correct farms. However, it will never be included in any final data products that are intended for sharing beyond your team, and no-one outside of your team will have access to it.')), - ]), - - Section::make(t('Other Farm Information')) - ->description(t('This section lets you add information about the farm that is not personally identifiable.')) - ->schema([ - KeyValue::make('properties') - ->hint(t('For example: gender of household head, active member of (name of your intervention project) - yes / no, farm typology information')) - ->helperText(t('The purpose of information here is to allow you to disaggregate results by these variables. For example, if you are interested in comparing results from farms that took part in a specific training activity with farms that did not take part, you should include that as a variable here. Variables entered here will be available in exported datasets so they can be used in your analysis.')), - ]), - - Section::make(t('GPS')) - ->description(t('Optionally, add the GPS co-ordinates for the farm')) - ->schema([ - TextInput::make('latitude') - ->label(t('Latitude')) - ->numeric() - ->minValue(-90) - ->maxValue(90), - TextInput::make('longitude') - ->label(t('Longitude')) - ->numeric() - ->minValue(-180) - ->maxValue(180), - TextInput::make('altitude') - ->label(t('Altitude')) - ->numeric() - ->minValue(-1240) - ->maxValue(60000), - TextInput::make('accuracy') - ->label(t('Accuracy')) - ->numeric(), - ])->columns(2), - - ]); - } - - public static function table(Table $table): Table - { - $farms = Farm::all()->where('owner_id', HelperService::getCurrentOwner()->id); - - $locationLevelColumns = $farms->map(fn(Farm $farm) => $farm->location->locationLevel) - ->unique() - ->values() - ->map( - fn(LocationLevel $locationLevel) => TextColumn::make("location_{$locationLevel->id}") - ->getStateUsing(fn($record) => $record->location->location_level_id === $locationLevel->id ? $record->location->name : '') - ->label($locationLevel->name) - ->sortable() - ->searchable() - ); - - $identifiers = $farms->map(fn(Farm $farm) => $farm->identifiers?->keys()) - ->flatten()->unique()->values(); - - $idColumns = $identifiers->map(fn($identifier) => TextColumn::make("identifiers.{$identifier}")->label(ucfirst($identifier))->sortable()->searchable()); - - $properties = $farms->map(fn(Farm $farm) => $farm->properties?->keys()) - ->flatten()->unique()->values(); - - $propertyColumns = $properties->map(fn($property) => TextColumn::make("properties.{$property}")->label(ucfirst($property))->sortable()->searchable()); - - return $table - ->columns([ - ...$locationLevelColumns, - TextColumn::make('team_code')->label(fn () => t('Unique code')) - ->sortable() - ->searchable(), - ...$idColumns, - ...$propertyColumns, - ]) - ->filters([]) - ->recordActions([ - EditAction::make(), - ]) - ->headerActions([ - CreateAction::make() - // disable New Farm button if there is no location level with farm - ->disabled(fn() => HelperService::getCurrentOwner()->locationLevels()->where('has_farms', 1)->count() < 1), - - // TODO: We have two location levels: district and sub-district. Can user select which location level when creating a new farm manually? - ]) - ->toolbarActions([ - BulkActionGroup::make([ - DeleteBulkAction::make(), - ]), - ]); - } - - public static function getRelations(): array - { - return [ - // - ]; - } - - public static function getPages(): array - { - return [ - 'index' => ListFarms::route('/'), - 'import' => ImportLocationsAndFarms::route('/import'), - ]; - } -} diff --git a/app/Filament/App/Clusters/LocationLevels/Resources/FarmResource/Pages/ImportLocationsAndFarms.php b/app/Filament/App/Clusters/LocationLevels/Resources/FarmResource/Pages/ImportLocationsAndFarms.php deleted file mode 100644 index e141751d..00000000 --- a/app/Filament/App/Clusters/LocationLevels/Resources/FarmResource/Pages/ImportLocationsAndFarms.php +++ /dev/null @@ -1,312 +0,0 @@ -disk ?: config('filesystems.default'); - } - - public function mount(): void - { - $this->form->fill(); - } - - // add a "Save Changes" button to submit the form - protected function getFormActions(): array - { - return [ - Action::make('save') - ->label(__('filament-panels::resources/pages/edit-record.form.actions.save.label')) - ->submit('save'), - ]; - } - - // add a function to handle the submitted form - public function save(): void - { - // get the submitted form data for further processing - $data = $this->form->getState(); - - // the uploaded excel file will be stored with import model for locations - // copy the uploaded excel file as a duplicate file, which will be stored with the import model for farms - Storage::copy($data['upload'], $data['upload'].'_duplicate'); - - // import locations - if ($data['override'] === 'yes') { - HelperService::getCurrentOwner()->locations()->delete(); - } - - $locationImport = Import::create([ - 'team_id' => HelperService::getCurrentOwner()->id, - 'model_type' => Location::class, - ]); - - $locationImport->addMedia(Storage::path($data['upload']))->toMediaCollection(); - $data['import_id'] = $locationImport->id; - - // import locations - Excel::import(new LocationImport($data), $locationImport->getFirstMediaPath()); - - // import farms - // create import record - for review and error tracking by users - $farmImport = Import::create([ - 'team_id' => HelperService::getCurrentOwner()->id, - 'model_type' => Farm::class, - ]); - - $farmImport->addMedia(Storage::path($data['upload']).'_duplicate')->toMediaCollection(); - $data['import_id'] = $farmImport->id; - - // import farms - Excel::import(new FarmImport($data), $farmImport->getFirstMediaPath()); - - // send notification - Notification::make() - ->title(t('Locations and farms are being imported.')) - ->body(t('The file will be processed in the background and the data will appear below once complete. You may leave this page without interrupting this process.')) - ->success() - ->send(); - - // redirect to farms list page - redirect(FarmResource::getUrl('index')); - } - - public function form(Schema $schema): Schema - { - return $schema - ->components([ - - Wizard::make([ - - // Step 1 - Step::make(t('Upload your farm list excel file')) - ->schema([ - - // Question: is the file upload works for ExcelImportAction's subclass only? - FileUpload::make('upload') - ->label(t('Location Levels and Farm List Excel Data')) - ->helperText(t('Please make sure your data is in the first worksheet of the Excel file, and that the first row contains the column headers.')) - ->disk($this->getDisk()) - ->columns() - ->required() - ->live() - ->preserveFilenames() - ->afterStateUpdated(function ($state, Set $set) { - // $state is only a TemporaryUploadedFile on the initial upload event; - // a later Livewire re-render of this field (e.g. after a validation - // error elsewhere in the form) passes back the already-stored path as - // a plain string instead - nothing new to parse in that case. - if (! $state instanceof TemporaryUploadedFile) { - return; - } - - $headings = (new HeadingRowImport)->toArray($state->getRealPath()); - - // $headings is an array(sheets) of arrays(headers) - // We only want the first sheet - $headings = $headings[0][0]; - - $set('header_columns', $headings ?? []); - }), - - ]), - - // Step 2 - Step::make(t('Map columns to location levels')) - ->schema( - - [ - Section::make(t('Column Mapping')) - ->columns(2) - ->schema(function ($livewire) { - - $hasFarmLevel = LocationLevel::where('has_farms', 1)->first(); - $currentLevel = $hasFarmLevel; - $parents = collect([]); - - while ($currentLevel->parent) { - $parents->push($currentLevel->parent); - $currentLevel = $currentLevel->parent; - } - - $parentQuestions = $parents->reverse()->map(callback: function ($parent) { - return collect([ - Select::make("parent_{$parent->id}_code_column") - ->label(t('Which column contains the').' '.$parent->name.' '.t('unique code?')) - ->options(fn (Get $get) => $get('header_columns')) - ->notIn(['na']) - ->required(), - Select::make("parent_{$parent->id}_name_column") - ->label(t('Which column contains the').' '.$parent->name.' '.t('name?')) - ->options(fn (Get $get) => $get('header_columns')) - ->notIn(['na']) - ->required(), - ]); - })->flatten(); - - $currentLevelQuestions = collect([ - Select::make('code_column') - ->label(t('Which column contains the').' '.$hasFarmLevel->name.' '.t('unique code?')) - ->options(fn (Get $get) => $get('header_columns')) - ->notIn(['na']) - ->required(), - Select::make('name_column') - ->label(t('Which column contains the').' '.$hasFarmLevel->name.' '.t('name?')) - ->options(fn (Get $get) => $get('header_columns')) - ->notIn(['na']) - ->required(), - ]); - - return $parentQuestions->merge($currentLevelQuestions)->toArray(); - }), - - Select::make('override') - ->label(t('Do you want to replace all locations with this import? (This will delete all existing locations from all location levels!)')) - ->options([ - 'no' => t('No'), - 'yes' => t('Yes'), - ]) - ->helperText(t('If you select "No", all existing locations will be kept. If you select "Yes", all existing locations will be deleted and replaced with the data from this import.')) - ->default('no'), - - Hidden::make('header_columns') - ->default(['na' => '~~upload a file to see the headers~~']) - ->live(), - - // find the location level that has farms - // Question: can one team has more than one location levels that have farms? - Hidden::make('level') - ->default(LocationLevel::where('has_farms', 1)->first()), - - Hidden::make('user_id') - ->default(fn () => auth()->id()), - - Hidden::make('owner_id') - ->default(HelperService::getCurrentOwner()->id), - ] - ), - - // Step 3 - Step::make(t('Map columns to farm')) - ->schema([ - - Hidden::make('header_columns') - ->default(['na' => '~~upload a file to see the column headers~~']) - ->live(), - - Section::make(t('Location')) - ->schema([ - - // Question: - // 1. can one team has more than one location levels that have farms? - // 2. can we set default value to simplify the import process? - Select::make('location_level_id') - ->label(t('Which location level are the farms linked to?')) - ->options( - LocationLevel::where('has_farms', true)->get()->pluck('name', 'id') - ) - ->placeholder(t('Select a location level')) - ->helperText(t('For many sampling strategies, this will be obvious (the lowest level). It may be less obvious when there are different hierarchies of locations in different places.')) - ->live(), - - Select::make('location_code_column') - ->options(fn (Get $get) => $get('header_columns')) - ->label(fn (Get $get) => t('Which column contains the').' '.(LocationLevel::find($get('location_level_id'))?->name ?? t('location')).' '.t('unique code?')) - ->placeholder(t('Select a column')), - ]), - - Section::make(t('Farm Information')) - ->columns(1) - ->schema([ - Select::make('farm_code_column') - ->label(t('Which column contains the farm unique code?')) - ->placeholder(t('Select a column')) - ->helperText(t('e.g. farm_id or farm_code')) - ->live() - ->options(fn (Get $get) => $get('header_columns')), - - CheckboxList::make('farm_identifiers') - ->label(t('Are there any additional columns that contain identifiers for the farm? Tick all that apply.')) - ->helperText(t('For example: family name, farm name, telephone numbers, etc. These are columns that can be useful for enumerators or project team members to identify the farm, but that should not be shared outside the project for data protection purposes.')) - ->options(fn (Get $get): array => $get('header_columns')) - ->disableOptionWhen( - fn (string $value, Get $get): bool => $value === (string) $get('farm_code_column') || - collect($get('farm_properties'))->contains($value) || - $value === 'na' - ) - ->live() - ->columnSpanFull(), - - CheckboxList::make('farm_properties') - ->label(t('Are there any additional columns that contain properties of the farm? Tick all that apply.')) - ->helperText(t('These are not identifiers, but are properties of the farm that are useful for analysis. For example: size of the farm, year of first engagement, etc. These are columns that can potentially be shared outside the project for analysis purposes.')) - ->options(fn (Get $get) => $get('header_columns')) - ->disableOptionWhen( - fn (string $value, Get $get): bool => $value === (string) $get('farm_code_column') || - collect($get('farm_identifiers'))->contains($value) || - $value === 'na' - ) - ->live() - ->columnSpanFull(), - - Hidden::make('owner_id') - ->default(HelperService::getCurrentOwner()->id), - - ]), - - Hidden::make('user_id') - ->default(auth()->id()), - - ]), - - ]), - - ])->statePath('data'); - } -} diff --git a/app/Filament/App/Clusters/LocationLevels/Resources/FarmResource/Pages/ListFarms.php b/app/Filament/App/Clusters/LocationLevels/Resources/FarmResource/Pages/ListFarms.php deleted file mode 100644 index 3280371d..00000000 --- a/app/Filament/App/Clusters/LocationLevels/Resources/FarmResource/Pages/ListFarms.php +++ /dev/null @@ -1,76 +0,0 @@ - t('Survey Dashboard'), - SurveyLocationsIndex::getUrl() => t('Survey locations'), - static::getUrl() => t('Farms'), - ]; - } - - protected function getHeaderWidgets(): array - { - return [ - FarmListHeaderWidget::class, - ]; - } - - protected function getHeaderActions(): array - { - return [ - - // add a button to divert to import custom page - Actions\Action::make('import') - ->label(fn () => t('Import Locations and Farm List')) - ->extraAttributes(['class' => 'buttonb']) - ->tooltip(fn () => t('Use this if you have your location and farm data all in one spreadsheet.')) - ->visible(fn () => auth()->user()->can('maintain list of farms')) - // disable if there is no location level with farms - ->disabled(fn() => HelperService::getCurrentOwner()->locationLevels()->where('has_farms', 1)->count() < 1) - ->url('farms/import'), - - ImportFarmsAction::make() - ->color('primary') - ->extraAttributes(['class' => 'buttonb']) - ->tooltip(fn () => t('Use this if you have already added your locations')) - ->visible(fn () => auth()->user()->can('maintain list of farms')) - // disable if there is no location level with farms - ->disabled(fn() => HelperService::getCurrentOwner()->locationLevels()->where('has_farms', 1)->count() < 1) - ->use(FarmImport::class) - ->label(fn () => t('Import Farm list')), - ]; - } - - #[On('echo:xlsforms,FarmImportCompleted')] - public function refreshTable(): void - { - $this->resetTable(); - } -} diff --git a/app/Filament/App/Clusters/LocationLevels/Resources/LocationLevelResource.php b/app/Filament/App/Clusters/LocationLevels/Resources/LocationLevelResource.php index 43cac177..8d7107db 100644 --- a/app/Filament/App/Clusters/LocationLevels/Resources/LocationLevelResource.php +++ b/app/Filament/App/Clusters/LocationLevels/Resources/LocationLevelResource.php @@ -59,9 +59,6 @@ public static function getNavigationItems(): array }); }); - // Points at the new ODK-Entities-backed Farms page - the original FarmResource is - // kept around (unlinked) for comparison during the migration. See - // docs/plans/odk-entities-farm-crud.md. $farmNavItem = NavigationItem::make('Farms') ->url(FarmEntityResource::getUrl()) ->isActiveWhen(fn () => request()->routeIs(FarmEntityResource::getRouteBaseName().'.index')); diff --git a/app/Filament/Tables/Actions/ImportFarmsAction.php b/app/Filament/Tables/Actions/ImportFarmsAction.php index ff1791cf..510855ac 100644 --- a/app/Filament/Tables/Actions/ImportFarmsAction.php +++ b/app/Filament/Tables/Actions/ImportFarmsAction.php @@ -3,7 +3,7 @@ namespace App\Filament\Tables\Actions; use App\Models\Import; -use App\Models\SampleFrame\Farm; +use App\Models\SampleFrame\FarmEntity; use App\Models\SampleFrame\LocationLevel; use App\Services\HelperService; use Closure; @@ -175,7 +175,7 @@ public function importData(): Closure // create import record - for review and error tracking by users $import = Import::create([ 'team_id' => HelperService::getCurrentOwner()->id, - 'model_type' => Farm::class, + 'model_type' => FarmEntity::class, ]); $import->addMedia(Storage::path($data['upload']))->toMediaCollection(); diff --git a/app/Imports/FarmEntityImport.php b/app/Imports/FarmEntityImport.php index b7ecb05e..6e5262ae 100644 --- a/app/Imports/FarmEntityImport.php +++ b/app/Imports/FarmEntityImport.php @@ -28,9 +28,8 @@ use Maatwebsite\Excel\Validators\ValidationException; /** - * ODK-Entities-backed counterpart to FarmImport: same column-mapping/validation rules, but - * rows are pushed to ODK Central (via OdkFarmEntityService::bulkCreateFarms()) instead of - * being inserted into a local farms table. + * Imports a farm list spreadsheet by pushing rows to ODK Central as entities (via + * OdkFarmEntityService::bulkCreateFarms()); only the FarmEntity stub rows are stored locally. * * WithMultipleSheets + sheets() => [0 => $this] restricts the import to the first worksheet * only while keeping all row-handling logic on this single class, delegating the sheet back diff --git a/app/Imports/FarmImport.php b/app/Imports/FarmImport.php deleted file mode 100644 index bb1f736d..00000000 --- a/app/Imports/FarmImport.php +++ /dev/null @@ -1,191 +0,0 @@ - [0 => $this] restricts the import to the first worksheet - * only while keeping all row-handling logic on this single class, delegating the sheet back - * to itself (mirrors Stats4sd\FilamentOdkLink\Imports\XlsformTemplate\XlsformModuleImport). - * - * This single-class shape is also what makes CSV files work. The PhpSpreadsheet Csv reader - * does not expose listWorksheetNames(), so maatwebsite/excel bypasses sheets() for CSV and - * applies THIS object as the row handler directly (see the "Csv doesn't have worksheets" - * branch of vendor/maatwebsite/excel/src/Reader.php::getWorksheets). Because collection() and - * the validation rules live here, CSV and Excel are handled by the same code. - */ -class FarmImport implements ShouldQueue, SkipsEmptyRows, ToCollection, WithCalculatedFormulas, WithChunkReading, WithEvents, WithHeadingRow, WithMultipleSheets, WithStrictNullComparison, WithValidation -{ - // The $data array is the data that is passed from the ImportFarmsAction form - public function __construct(public array $data) {} - - public function sheets(): array - { - return [ - 0 => $this, - ]; - } - - public function collection(Collection $rows): array - { - $importedFarms = []; - - foreach ($rows as $row) { - $headers = $this->data['header_columns']; - - $farmCodeColumn = $headers[$this->data['farm_code_column']]; - $locationLevel = LocationLevel::find($this->data['location_level_id']); - $locationCodeColumn = $headers[$this->data['location_code_column']]; - $location = Location::where('code', $row[$locationCodeColumn]) - ->where('location_level_id', $locationLevel->id) - ->first(); - - // Find the identifier columns; - $identifierColumns = collect($this->data['farm_identifiers'])->map(fn ($identifier) => $headers[$identifier]); - // Get the data from those columns; - $identifierData = $identifierColumns->mapWithKeys(fn ($column) => [$column => $row[$column]]); - - // Find the property columns; - $propertyColumns = collect($this->data['farm_properties'])->map(fn ($property) => $headers[$property]); - // Get the data from those columns; - $propertyData = $propertyColumns->mapWithKeys(fn ($column) => [$column => $row[$column]]); - - // check if farm with unique code existed in this team. - // it is not advised to use upsert here. The old farm and new farm with unique code could be two different farms. - // human intervention is required to handle this sitation. - // If they are two different farms, this can be resolved by assigning a new unique code to the new farm. - $noOfRecords = Farm::where('owner_id', $this->data['owner_id'])->where('team_code', $row[$farmCodeColumn])->count(); - - // only create farms record if unique code is not existed for this team - if ($noOfRecords == 0) { - // Create the farm - $farm = new Farm([ - 'owner_id' => $this->data['owner_id'], - 'location_id' => $location->id, - 'team_code' => $row[$farmCodeColumn], - 'identifiers' => $identifierData, - 'properties' => $propertyData, - ]); - $farm->save(); - - $importedFarms[] = $farm; - } - } - - return $importedFarms; - } - - public function rules(): array - { - $headers = $this->data['header_columns']; - $locationCodeColumn = $headers[$this->data['location_code_column']]; - $farmCodeColumn = $headers[$this->data['farm_code_column']]; - - return [ - $locationCodeColumn => 'required|exists:locations,code', - $farmCodeColumn => 'required', - ]; - } - - public function customValidationMessages(): array - { - $headers = $this->data['header_columns']; - $locationCodeColumn = $headers[$this->data['location_code_column']]; - $farmCodeColumn = $headers[$this->data['farm_code_column']]; - - return [ - "$locationCodeColumn.required" => "The $locationCodeColumn cannot be empty.", - "$locationCodeColumn.exists" => 'The location with this code does not exist in the database.', - "$farmCodeColumn.required" => 'The farm code cannot be empty.', - ]; - } - - public function chunkSize(): int - { - return 1000; - } - - public function registerEvents(): array - { - return [ - ImportFailed::class => function (ImportFailed $event) { - - // check if exception is a validation exception, and get the failures from it - if ($event->getException() instanceof ValidationException) { - $failures = collect($event->getException()->failures()); - - Import::find($this->data['import_id']) - ->update([ - 'errors' => $failures->map(function ($failure) { - return [ - 'location' => [ - 'row' => $failure->row(), - 'column' => $failure->attribute(), - ], - 'errors' => $failure->errors(), - ]; - })->toArray(), - ]); - } else { - Import::find($this->data['import_id']) - ->update([ - 'errors' => [ - [ - 'row' => null, - 'attribute' => null, - 'errors' => [$event->getException()->getMessage()], - ], - ], - ]); - } - - $recipient = User::find($this->data['user_id']); - - Notification::make() - ->title('Import of Farm Data Failed') - ->body(fn (): HtmlString => new HtmlString( - 'The import of farm data failed with the following errors:

' - .$event->getException()->getMessage() - )) - ->danger() - ->sendToDatabase($recipient, isEventDispatched: true) - ->broadcast($recipient); - }, - AfterImport::class => function (AfterImport $event) { - - $recipient = User::find($this->data['user_id']); - - // send notification - Notification::make() - ->title('Import of Farm Data Complete') - ->success() - ->sendToDatabase($recipient, isEventDispatched: true) - ->broadcast($recipient); - }, - ]; - } -} diff --git a/app/Livewire/DataCollection/DataCollectionByFarm.php b/app/Livewire/DataCollection/DataCollectionByFarm.php deleted file mode 100644 index c159c023..00000000 --- a/app/Livewire/DataCollection/DataCollectionByFarm.php +++ /dev/null @@ -1,67 +0,0 @@ -team->locationLevels()->where('has_farms', true)->first(); - - return $table - ->heading('Farms') - ->relationship(fn() => $this->team->farms()) - ->filters([ - SelectFilter::make('parent_' . $locationLevel->id) - ->label('Location') - ->relationship('location', 'name', fn($query) => $query->where('location_level_id', $locationLevel->id)), - ]) - ->columns([ - ColumnGroup::make('Location', [ - TextColumn::make('location.name')->label(Str::of($locationLevel->name)->title()), - TextColumn::make('identifying_attribute')->label('Farm Name'), - ]), - ColumnGroup::make('Surveys Completed', [ - IconColumn::make('household_form_completed')->boolean(), - IconColumn::make('fieldwork_form_completed')->boolean(), - ]) - ]); - - } -} diff --git a/app/Livewire/DataCollection/DataCollectionByLocation.php b/app/Livewire/DataCollection/DataCollectionByLocation.php index 75608461..48351344 100644 --- a/app/Livewire/DataCollection/DataCollectionByLocation.php +++ b/app/Livewire/DataCollection/DataCollectionByLocation.php @@ -13,17 +13,16 @@ use Filament\Tables\Columns\TextColumn; use Filament\Tables\Concerns\InteractsWithTable; use Filament\Tables\Contracts\HasTable; -use Filament\Tables\Filters\Filter; use Filament\Tables\Filters\SelectFilter; +use Filament\Tables\Table; use Illuminate\Database\Eloquent\Builder; use Illuminate\Support\Str; use Livewire\Attributes\Computed; use Livewire\Attributes\Reactive; use Livewire\Attributes\Url; use Livewire\Component; -use Filament\Tables\Table; -class DataCollectionByLocation extends Component implements HasTable, HasForms, HasActions +class DataCollectionByLocation extends Component implements HasActions, HasForms, HasTable { use InteractsWithActions; use InteractsWithForms; @@ -37,48 +36,40 @@ class DataCollectionByLocation extends Component implements HasTable, HasForms, #[Url] public ?array $tableFilters = null; - #[Computed] public function locationLevel() { return LocationLevel::find($this->locationLevelId); } - public function table(Table $table): Table { $parentLocationLevel = $this->locationLevel?->parent; $filters = []; if ($parentLocationLevel) { - $filters[] = SelectFilter::make('parent_' . $parentLocationLevel->id) - ->relationship('parent', 'name', fn(Builder $query) => $query->where('location_level_id', $parentLocationLevel->id)); + $filters[] = SelectFilter::make('parent_'.$parentLocationLevel->id) + ->relationship('parent', 'name', fn (Builder $query) => $query->where('location_level_id', $parentLocationLevel->id)); } return $table - ->relationship(fn() => $this->locationLevel->locations()) - ->heading(fn() => Str::of($this->locationLevel->name)->title()->plural()) + ->relationship(fn () => $this->locationLevel->locations()) + ->heading(fn () => Str::of($this->locationLevel->name)->title()->plural()) ->filters($filters) ->columns([ ColumnGroup::make('Location', [ TextColumn::make('name')->label('') - ->url(fn(Location $record) => MonitorDataCollection::getUrl() . '?' . http_build_query([ - 'locationLevelId' => $record->locationLevel->children->first()?->id, - 'tableFilters' => [ - 'parent_' . $record->locationLevel->id => ['value' => $record->id], - ], - ])), - ]), - ColumnGroup::make('Farm Counts', [ - TextColumn::make('farms_all_count')->label('Total'), - TextColumn::make('farms_household_complete_count')->label('Household Complete'), - TextColumn::make('farms_fieldwork_complete_count')->label('Fieldwork Complete'), - TextColumn::make('farms_all_complete_count')->label('All Complete'), + ->url(fn (Location $record) => MonitorDataCollection::getUrl().'?'.http_build_query([ + 'locationLevelId' => $record->locationLevel->children->first()?->id, + 'tableFilters' => [ + 'parent_'.$record->locationLevel->id => ['value' => $record->id], + ], + ])), ]), + TextColumn::make('farms_all_count')->label('Farms'), ]); } - public function render() { return view('livewire.data-collection.data-collection-by-location'); diff --git a/app/Models/Interfaces/RepeatModel.php b/app/Models/Interfaces/RepeatModel.php deleted file mode 100644 index cb37a4c8..00000000 --- a/app/Models/Interfaces/RepeatModel.php +++ /dev/null @@ -1,17 +0,0 @@ - 'collection', - 'properties' => 'collection', - 'household_form_completed' => 'boolean', - 'fieldwork_form_completed' => 'boolean', - 'refused' => 'boolean', - ]; - - protected static function booted() - { - static::saved(function (self $location) { - $location->owner->xlsforms()->update(['draft_needs_update' => true]); - }); - } - - /** @return BelongsTo */ - public function owner(): BelongsTo - { - return $this->belongsTo(Team::class); - } - - public function getCsvContentsForOdk(?WithXlsforms $team = null): array - { - return [ - 'id' => $this->id, - 'location_id' => $this->location_id, - 'location_name' => $this->location?->name, - 'team_code' => $this->team_code, - 'team_code_name' => $this->identifiers ? $this->identifiers['name'] . '(No. ' . $this->team_code . ')' : 'No. ' . $this->team_code, - 'name' => $this->identifiers ? $this->identifiers['name'] : '', - 'sex' => $this->properties ? $this->properties['sex'] : '', - 'year' => $this->properties ? $this->properties['year'] : '', - 'reserve' => $this->identifiers && $this->identifiers->has('reserve') ? $this->identifiers['reserve'] : '', // value is 0 = beneficiary farm that is not a reserve; 1 = beneficiary farm that is a reserve; '' = non-beneficiary farm. - ]; - } - - public function location(): BelongsTo - { - return $this->belongsTo(Location::class); - } - - public function farmSurveyData(): HasMany - { - return $this->hasMany(FarmSurveyData::class); - } - - /** @return Attribute */ - protected function identifyingAttribute(): Attribute - { - return new Attribute( - get: fn() => $this->identifiers['name'] ?? ($this->identifiers->first() ?? null), - ); - } - - public function updateCompletionStatus(): void - { - // check pilot completion - $this->submissions - ->filter(fn(Submission $submission) => $submission->test_data) - ->each(function (Submission $submission) { - - if (Str::contains($submission->xlsformVersion->xlsform->xlsformTemplate->title, 'HOLPA Household Form')) { - $this->household_pilot_completed = true; - } - - if (Str::contains($submission->xlsformVersion->xlsform->xlsformTemplate->title, 'HOLPA Fieldwork Form')) { - $this->fieldwork_pilot_completed = true; - } - - $this->save(); - }); - - $this->submissions - ->filter(fn(Submission $submission) => !$submission->test_data) - ->each(function (Submission $submission) { - - if (Str::contains($submission->xlsformVersion->xlsform->xlsformTemplate->title, 'HOLPA Household Form')) { - $this->household_form_completed = true; - } - - if (Str::contains($submission->xlsformVersion->xlsform->xlsformTemplate->title, 'HOLPA Fieldwork Form')) { - $this->fieldwork_form_completed = true; - } - - $this->save(); - }); - - - } -} diff --git a/app/Models/SampleFrame/Location.php b/app/Models/SampleFrame/Location.php index 3536782c..ef7c95e2 100644 --- a/app/Models/SampleFrame/Location.php +++ b/app/Models/SampleFrame/Location.php @@ -53,11 +53,6 @@ public function children(): HasMany return $this->hasMany(self::class, 'parent_id'); } - public function farms(): HasMany - { - return $this->hasMany(Farm::class); - } - public function farmEntities(): HasMany { return $this->hasMany(FarmEntity::class); @@ -88,48 +83,6 @@ public function farmsAllCount(): Attribute ); } - public function farmsHouseholdCompleteCount(): Attribute - { - return new Attribute( - get: function () { - return Cache::remember($this->cacheKey().':farmsHouseholdCompleteCount', now()->addMinutes(5), function () { - return $this->children->reduce(function ($carry, $location) { - return $carry + $location->farms_household_complete_count; - }, $this->farms()->where('household_form_completed', true)->count()); - }); - } - ); - } - - public function farmsFieldworkCompleteCount(): Attribute - { - return new Attribute( - get: function () { - return Cache::remember($this->cacheKey().':farmsFieldworkCompleteCount', now()->addMinutes(5), function () { - return $this->children->reduce(function ($carry, $location) { - return $carry + $location->farms_fieldwork_complete_count; - }, $this->farms()->where('fieldwork_form_completed', true)->count()); - }); - } - ); - } - - public function farmsAllCompleteCount(): Attribute - { - return new Attribute( - get: function () { - return Cache::remember($this->cacheKey().':farmsAllCompleteCount', now()->addMinutes(5), function () { - return $this->children->reduce(function ($carry, $location) { - return $carry + $location->farms_all_complete_count; - }, $this->farms() - ->where('household_form_completed', true) - ->where('fieldwork_form_completed', true) - ->count()); - }); - } - ); - } - // from https://laravel-news.com/laravel-model-caching public function cacheKey(): string { diff --git a/app/Models/SurveyData/Crop.php b/app/Models/SurveyData/Crop.php deleted file mode 100644 index 93aee518..00000000 --- a/app/Models/SurveyData/Crop.php +++ /dev/null @@ -1,27 +0,0 @@ - 'collection', - ]; - - public function farmSurveyData(): BelongsTo - { - return $this->belongsTo(FarmSurveyData::class, 'submission_id', 'submission_id'); - } - - public function submission(): BelongsTo - { - return $this->belongsTo(Submission::class, 'submission_id', 'id'); - } -} diff --git a/app/Models/SurveyData/FarmSurveyData.php b/app/Models/SurveyData/FarmSurveyData.php deleted file mode 100644 index 4a26eb78..00000000 --- a/app/Models/SurveyData/FarmSurveyData.php +++ /dev/null @@ -1,43 +0,0 @@ - 'collection', - ]; - - public function submission(): BelongsTo - { - return $this->belongsTo(Submission::class, 'submission_id', 'id'); - } - - public function farm(): BelongsTo - { - return $this->belongsTo(Farm::class); - } - - // Link to repeat groups - - public function crops(): HasMany - { - return $this->hasMany(Crop::class, 'farm_survey_data_id', 'id'); - } - - public function livestocks(): HasMany - { - return $this->hasMany(Livestock::class, 'farm_survey_data_id', 'id'); - } - -} diff --git a/app/Models/SurveyData/Livestock.php b/app/Models/SurveyData/Livestock.php deleted file mode 100644 index 2e9c9662..00000000 --- a/app/Models/SurveyData/Livestock.php +++ /dev/null @@ -1,28 +0,0 @@ - 'collection', - ]; - - public function farmSurveyData(): BelongsTo - { - return $this->belongsTo(FarmSurveyData::class, 'submission_id', 'submission_id'); - } - - public function submission(): BelongsTo - { - return $this->belongsTo(Submission::class, 'submission_id', 'id'); - } -} diff --git a/app/Models/Team.php b/app/Models/Team.php index f3223025..93084bec 100644 --- a/app/Models/Team.php +++ b/app/Models/Team.php @@ -2,7 +2,6 @@ namespace App\Models; -use App\Models\SampleFrame\Farm; use App\Models\SampleFrame\Location; use App\Models\SampleFrame\LocationLevel; use App\Services\XlsformModules\FarmInfoModuleBuilder; @@ -146,12 +145,6 @@ public function locations(): HasMany return $this->hasMany(Location::class, 'owner_id'); } - /** @return HasMany */ - public function farms(): HasMany - { - return $this->hasMany(Farm::class, 'owner_id'); - } - /** @return HasMany */ public function imports(): HasMany { @@ -274,8 +267,7 @@ protected function pilotProgress(): Attribute return 'complete'; } - // $farm->household_form_completed + fieldwork_form_completed are only marked for 'live' submissions, so here we can just count if any submissions have come in. - if ($this->farms->some(fn (Farm $farm) => $farm->submissions()->count() > 0)) { + if ($this->xlsforms()->whereHas('xlsformVersions.submissions')->exists()) { return 'in_progress'; } @@ -307,7 +299,7 @@ protected function dataCollectionProgress(): Attribute return 'complete'; } - if ($this->farms->some(fn (Farm $farm) => $farm->household_form_completed || $farm->fieldwork_form_completed)) { + if ($this->xlsforms()->whereHas('xlsformVersions.submissions', fn ($query) => $query->where('test_data', false))->exists()) { return 'in_progress'; } @@ -330,7 +322,7 @@ protected function shouldReceiveAllXlsformTemplates(): Attribute public function readyForLive(): Attribute { return new Attribute( - get: fn(): bool => $this->languages_complete && $this->sampling_complete && $this->pba_complete && $this->optional_modules_complete, + get: fn (): bool => $this->languages_complete && $this->sampling_complete && $this->pba_complete && $this->optional_modules_complete, ); } @@ -351,7 +343,7 @@ public function deployDraftForms(): void $xlsformsToUpdate = $this->xlsforms->filter(fn (Xlsform $xlsform) => $xlsform->draft_needs_update); - if($xlsformsToUpdate->count() === 0) { + if ($xlsformsToUpdate->count() === 0) { return; } diff --git a/app/Policies/FarmEntityPolicy.php b/app/Policies/FarmEntityPolicy.php index 00e6570f..66c15d86 100644 --- a/app/Policies/FarmEntityPolicy.php +++ b/app/Policies/FarmEntityPolicy.php @@ -5,8 +5,6 @@ use App\Models\SampleFrame\FarmEntity; use App\Models\User; -// Mirrors FarmPolicy's gate names so the same permissions govern both the legacy -// database-backed Farms page and this new ODK-Entities-backed one during coexistence. class FarmEntityPolicy { public function viewAny(User $user): bool diff --git a/app/Policies/FarmPolicy.php b/app/Policies/FarmPolicy.php deleted file mode 100644 index 9a0dc9a6..00000000 --- a/app/Policies/FarmPolicy.php +++ /dev/null @@ -1,49 +0,0 @@ -can('view list of farms'); - } - - public function view(User $user, Farm $farm): bool - { - return $user->can('view list of farms'); - } - - public function create(User $user): bool - { - return $user->can('maintain list of farms'); - } - - public function update(User $user, Farm $farm): bool - { - return $user->can('maintain list of farms'); - } - - public function delete(User $user, Farm $farm): bool - { - return $user->can('maintain list of farms'); - } - - public function deleteAny(User $user): bool - { - return $user->can('maintain list of farms'); - } - - public function restore(User $user, Farm $farm): bool - { - return $user->can('maintain list of farms'); - } - - public function forceDelete(User $user, Farm $farm): bool - { - return $user->can('maintain list of farms'); - } -} diff --git a/app/Providers/AppServiceProvider.php b/app/Providers/AppServiceProvider.php index 81eea255..e885f47a 100644 --- a/app/Providers/AppServiceProvider.php +++ b/app/Providers/AppServiceProvider.php @@ -5,7 +5,6 @@ use App\Models\Holpa\Domain; use App\Models\Holpa\GlobalIndicator; use App\Models\Holpa\Theme; -use App\Models\SampleFrame\Farm; use App\Models\SampleFrame\FarmEntity; use App\Models\SampleFrame\LocationLevel; use App\Models\Team; @@ -15,7 +14,6 @@ use App\Policies\DatasetVariablePolicy; use App\Policies\DomainPolicy; use App\Policies\FarmEntityPolicy; -use App\Policies\FarmPolicy; use App\Policies\GlobalIndicatorPolicy; use App\Policies\LocationLevelPolicy; use App\Policies\ProgramPolicy; @@ -75,7 +73,6 @@ public function boot(): void Gate::policy(Dataset::class, DatasetPolicy::class); Gate::policy(DatasetVariable::class, DatasetVariablePolicy::class); Gate::policy(LocationLevel::class, LocationLevelPolicy::class); - Gate::policy(Farm::class, FarmPolicy::class); Gate::policy(FarmEntity::class, FarmEntityPolicy::class); // Enable migrations in subfolders diff --git a/app/Services/HelperService.php b/app/Services/HelperService.php index 2ae2aa79..baec0ade 100644 --- a/app/Services/HelperService.php +++ b/app/Services/HelperService.php @@ -2,52 +2,22 @@ namespace App\Services; -use App\Models\SampleFrame\Farm; use App\Models\Team; use Filament\Facades\Filament; -use Illuminate\Database\Eloquent\Model; -use Illuminate\Support\Str; class HelperService { // Get the current team with the correct namespacing so phpstan doesn't complain whenever we get the current team - public static function getCurrentOwner(): Team|null + public static function getCurrentOwner(): ?Team { if (Filament::hasTenancy() && is_a(Filament::getTenant(), Team::class)) { /** @var Team $team */ $team = Filament::getTenant(); + return $team; } return null; } - - // find farm's full location details for data export to Excel file - public static function findFarmLocationDetails(string $farmId): array - { - // find farm model - $farm = Farm::find($farmId); - - // array for storing farm and location levels details - $array = []; - - $array['farm_name'] = $farm->identifiers['name']; - $array['farm_id'] = $farm->id; - - $tempLocation = $farm->location; - - // find all location levels until there is no more parent location - do { - // a location ID field name should be lowercase, with underscore instead of hypen, e.g. sub_district - $locationIdFieldName = Str::lower(Str::replace('-', '_', $tempLocation->locationLevel->name)); - - $array[$locationIdFieldName.'_name'] = $tempLocation->name; - $array[$locationIdFieldName.'_id'] = $tempLocation->id; - - $tempLocation = $tempLocation->parent; - } while ($tempLocation != null); - - return array_reverse($array); - } } diff --git a/app/Services/OdkFarmEntityService.php b/app/Services/OdkFarmEntityService.php index 247c4b3e..125ad48a 100644 --- a/app/Services/OdkFarmEntityService.php +++ b/app/Services/OdkFarmEntityService.php @@ -555,7 +555,7 @@ public function createFarm( * Bulk-creates many farms in a single Central API call - used by the Excel import * flow instead of calling createFarm() per row, which would mean one Central round * trip per row on top of the reconciliation calls. Rows whose team_code already - * exists for the team are skipped (mirrors the FarmImport dedup rule); a + * exists for the team are skipped; a * team_code repeated within $rows itself is also deduped, keeping the first occurrence. * * Each row: locationId (int), teamCode (string), identifiers (array), @@ -828,8 +828,8 @@ public function updateFarm( /** * Soft-deletes a farm's entity on ODK Central, then soft-deletes the local FarmEntity - * row (not hard-deleted - keeps it around for any future FK references, e.g. from - * FarmSurveyData once this is wired up at cutover). + * row (not hard-deleted - keeps it around for any future FK references from submission + * data, once submissions are linked to the farm they describe). */ public function deleteFarm(FarmEntity $farmEntity): void { diff --git a/database/migrations/06_locations/2024_06_11_06_104524_create_farms_table.php b/database/migrations/06_locations/2024_06_11_06_104524_create_farms_table.php deleted file mode 100644 index b218b31a..00000000 --- a/database/migrations/06_locations/2024_06_11_06_104524_create_farms_table.php +++ /dev/null @@ -1,50 +0,0 @@ -id(); - $table->foreignId('owner_id')->constrained('teams')->cascadeOnDelete()->cascadeOnUpdate(); - $table->foreignId('location_id')->constrained()->cascadeOnDelete()->cascadeOnUpdate(); - $table->string('team_code'); // should be unique per team - - // identifiers - $table->json('identifiers')->nullable(); // identifiers - by default, we consider this personally identifying information of the farm. - - // location - $table->decimal('latitude', 11, 8)->nullable(); - $table->decimal('longitude', 11, 8)->nullable(); - $table->integer('altitude')->nullable(); - $table->decimal('accuracy', 9, 4)->nullable(); - - // Probably temporary until we figure out a generalised approach - $table->boolean('household_form_completed')->default(false)->comment('Is household form completed for this farm?'); - $table->boolean('fieldwork_form_completed')->default(false)->comment('Is fieldwork form completed for this farm?'); - - $table->boolean('household_pilot_completed')->default(false)->comment('Is household form completed for this farm with a pilot-test submission?'); - $table->boolean('fieldwork_pilot_completed')->default(false)->comment('Is fieldwork form completed for this farm with a pilot test submission?'); - - $table->boolean('refused')->default(false); - - $table->json('properties')->nullable(); // other properties; - - $table->timestamps(); - }); - } - - /** - * Reverse the migrations. - */ - public function down(): void - { - Schema::dropIfExists('farms'); - } -}; diff --git a/database/migrations/07_survey_data/2024_08_12_11_121716_create_farm_survey_data_table.php b/database/migrations/07_survey_data/2024_08_12_11_121716_create_farm_survey_data_table.php deleted file mode 100644 index 74912f36..00000000 --- a/database/migrations/07_survey_data/2024_08_12_11_121716_create_farm_survey_data_table.php +++ /dev/null @@ -1,59 +0,0 @@ - 8126). Changing some columns to TEXT or BLOB may help. In current row format, BLOB prefix of 0 bytes is stored inline." - - // Google searched below stackoverflow thread: - // MySQL: Error Code: 1118 Row size too large (> 8126). Changing some columns to TEXT or BLOB - // https://stackoverflow.com/questions/22637733/mysql-error-code-1118-row-size-too-large-8126-changing-some-columns-to-te - - // Solution: - // Run below SQL in TablePlus - // SET GLOBAL innodb_strict_mode = 0; - - /** - * Run the migrations. - */ - public function up(): void - { - Schema::create('farm_survey_data', function (Blueprint $table) { - $table->id(); - - $table->json('properties')->nullable(); - - $table->text('start')->nullable(); - $table->text('end')->nullable(); - $table->text('today')->nullable(); - $table->text('deviceid')->nullable(); - $table->text('inquirer')->nullable(); - - $table->text('household_survey_date')->nullable(); - $table->unsignedBigInteger('farm_id')->nullable(); - - // location - $table->decimal('latitude', 11, 8)->nullable(); - $table->decimal('longitude', 11, 8)->nullable(); - $table->integer('altitude')->nullable(); - $table->decimal('accuracy', 9, 4)->nullable(); - $table->text('gps_location_alt')->nullable(); - - $table->foreignId('submission_id')->nullable(); - - $table->timestamps(); - }); - } - - /** - * Reverse the migrations. - */ - public function down(): void - { - Schema::dropIfExists('farm_survey_data'); - } -}; diff --git a/database/migrations/07_survey_data/2024_08_12_12_111301_create_crops_table.php b/database/migrations/07_survey_data/2024_08_12_12_111301_create_crops_table.php deleted file mode 100644 index 29aea215..00000000 --- a/database/migrations/07_survey_data/2024_08_12_12_111301_create_crops_table.php +++ /dev/null @@ -1,134 +0,0 @@ -id(); - - $table->foreignId('farm_survey_data_id')->nullable(); - $table->json('properties')->nullable(); - - $table->text('primary_crop_id')->nullable(); - $table->text('primary_crop_name')->nullable(); - $table->integer('crop_varities')->nullable(); - $table->decimal('primary_crop_area', 24, 6)->nullable(); - $table->decimal('primary_crop_area_ha', 24, 6)->nullable(); - $table->text('primary_crop_practices')->nullable(); - - $table->decimal('area_annual_mono', 24, 6)->nullable(); - $table->decimal('area_perennial_mono', 24, 6)->nullable(); - $table->decimal('area_afroforestry', 24, 6)->nullable(); - $table->decimal('area_burning', 24, 6)->nullable(); - $table->decimal('area_cover', 24, 6)->nullable(); - $table->decimal('area_rotation', 24, 6)->nullable(); - $table->decimal('area_fallow', 24, 6)->nullable(); - $table->decimal('area_hedgerows', 24, 6)->nullable(); - $table->decimal('area_garden', 24, 6)->nullable(); - $table->decimal('area_intercrop', 24, 6)->nullable(); - $table->decimal('area_landclearing', 24, 6)->nullable(); - $table->decimal('area_mulch', 24, 6)->nullable(); - $table->decimal('area_natural', 24, 6)->nullable(); - $table->decimal('area_pollinator', 24, 6)->nullable(); - $table->decimal('area_push_pull', 24, 6)->nullable(); - $table->decimal('area_other_practice', 24, 6)->nullable(); - - $table->decimal('density_annual_mono', 20, 2)->nullable(); - $table->decimal('density_perennial_mono', 20, 2)->nullable(); - $table->decimal('density_afroforestry', 20, 2)->nullable(); - $table->decimal('density_burning', 20, 2)->nullable(); - $table->decimal('density_cover', 20, 2)->nullable(); - $table->decimal('density_rotation', 20, 2)->nullable(); - $table->decimal('density_fallow', 20, 2)->nullable(); - $table->decimal('density_hedgerows', 20, 2)->nullable(); - $table->decimal('density_garden', 20, 2)->nullable(); - $table->decimal('density_intercrop', 20, 2)->nullable(); - $table->decimal('density_landclearing', 20, 2)->nullable(); - $table->decimal('density_mulch', 20, 2)->nullable(); - $table->decimal('density_natural', 20, 2)->nullable(); - $table->decimal('density_pollinator', 20, 2)->nullable(); - $table->decimal('density_push_pull', 20, 2)->nullable(); - $table->decimal('density_other_practice', 20, 2)->nullable(); - - $table->text('agroforestry_crops')->nullable(); - $table->text('cover_crops')->nullable(); - $table->text('rotation_crops')->nullable(); - $table->text('garden_crops')->nullable(); - $table->text('intercrop_crops')->nullable(); - $table->text('push_pull_crops')->nullable(); - $table->text('agroforestry_trees_n')->nullable(); - $table->text('hedgerows_trees_n')->nullable(); - $table->text('homegarden_trees_n')->nullable(); - $table->text('agroforestry_trees_diversity')->nullable(); - $table->text('hedgerows_trees_diversity')->nullable(); - $table->text('homegarden_trees_diversity')->nullable(); - $table->text('agroforestry_trees_spatial')->nullable(); - $table->text('hedgerows_trees_spatial')->nullable(); - $table->text('homegarden_trees_spatial')->nullable(); - $table->text('yield_unit')->nullable(); - - $table->text('yield_unit_kg_conversion')->nullable(); - $table->text('yield_unit_label')->nullable(); - $table->text('yield_unit_label_english')->nullable(); - $table->text('total_yield')->nullable(); - $table->text('yield_kg')->nullable(); - $table->text('yield_weight_area')->nullable(); - $table->text('yield_kg_ha')->nullable(); - - $table->decimal('yield_annual_mono', 20, 2)->nullable(); - $table->decimal('yield_perennial_mono', 20, 2)->nullable(); - $table->decimal('yield_afroforestry', 20, 2)->nullable(); - $table->decimal('yield_burning', 20, 2)->nullable(); - $table->decimal('yield_cover', 20, 2)->nullable(); - $table->decimal('yield_rotation', 20, 2)->nullable(); - $table->decimal('yield_fallow', 20, 2)->nullable(); - $table->decimal('yield_hedgerows', 20, 2)->nullable(); - $table->decimal('yield_garden', 20, 2)->nullable(); - $table->decimal('yield_intercrop', 20, 2)->nullable(); - $table->decimal('yield_landclearing', 20, 2)->nullable(); - $table->decimal('yield_mulch', 20, 2)->nullable(); - $table->decimal('yield_natural', 20, 2)->nullable(); - $table->decimal('yield_pollinator', 20, 2)->nullable(); - $table->decimal('yield_push_pull', 20, 2)->nullable(); - $table->decimal('yield_other_practice', 20, 2)->nullable(); - - $table->text('primary_crop_use')->nullable(); - $table->text('primary_crop_use_other')->nullable(); - $table->text('yield_sums')->nullable(); - - $table->decimal('amount_own_consumption', 20, 2)->nullable(); - $table->decimal('amount_livestock_consumption', 20, 2)->nullable(); - $table->decimal('amount_consumer', 20, 2)->nullable(); - $table->decimal('amount_coop', 20, 2)->nullable(); - $table->decimal('amount_market', 20, 2)->nullable(); - $table->decimal('amount_trader', 20, 2)->nullable(); - $table->decimal('amount_gift', 20, 2)->nullable(); - $table->decimal('amount_wasted', 20, 2)->nullable(); - $table->decimal('amount_other_use', 20, 2)->nullable(); - $table->decimal('consumer_price', 20, 2)->nullable(); - $table->decimal('coop_price', 20, 2)->nullable(); - $table->decimal('market_price', 20, 2)->nullable(); - $table->decimal('trader_price', 20, 2)->nullable(); - - $table->foreignId('submission_id')->nullable(); - - $table->timestamps(); - }); - } - - /** - * Reverse the migrations. - */ - public function down(): void - { - Schema::dropIfExists('crops'); - } -}; diff --git a/database/migrations/07_survey_data/2024_08_12_12_111310_create_livestocks_table.php b/database/migrations/07_survey_data/2024_08_12_12_111310_create_livestocks_table.php deleted file mode 100644 index e4337b80..00000000 --- a/database/migrations/07_survey_data/2024_08_12_12_111310_create_livestocks_table.php +++ /dev/null @@ -1,43 +0,0 @@ -id(); - - $table->foreignId('farm_survey_data_id')->nullable(); - $table->json('properties')->nullable(); - - $table->unsignedInteger('livestock_id')->nullable(); - $table->text('livestock_name')->nullable(); - $table->text('livestock_other_check')->nullable(); - $table->text('livestock_label')->nullable(); - $table->integer('livestock_breeds')->nullable(); - $table->integer('number_raised')->nullable(); - $table->text('livestock_filter')->nullable(); - $table->text('livestock_uses')->nullable(); - $table->text('livestock_use_other')->nullable(); - - $table->foreignId('submission_id')->nullable(); - - $table->timestamps(); - }); - } - - /** - * Reverse the migrations. - */ - public function down(): void - { - Schema::dropIfExists('livestocks'); - } -}; diff --git a/database/migrations/2026_07_31_000001_drop_legacy_farm_tables.php b/database/migrations/2026_07_31_000001_drop_legacy_farm_tables.php new file mode 100644 index 00000000..4ac21731 --- /dev/null +++ b/database/migrations/2026_07_31_000001_drop_legacy_farm_tables.php @@ -0,0 +1,19 @@ +delete(); // datasets mentioned in Data Structure excel file - $farmSurveyDataset = Dataset::create(['name' => 'Farm Survey Data', 'parent_id' => NULL, 'primary_key' => 'id', 'entity_model' => FarmSurveyData::class]); - Dataset::create(['name' => 'Crops', 'parent_id' => $farmSurveyDataset->id, 'primary_key' => 'id', 'entity_model' => Crop::class]); - Dataset::create(['name' => 'Farms', 'parent_id' => $farmSurveyDataset->id, 'primary_key' => 'id', 'entity_model' => Farm::class]); - Dataset::create(['name' => 'Livestock', 'parent_id' => $farmSurveyDataset->id, 'primary_key' => 'id', 'entity_model' => Livestock::class]); + $farmSurveyDataset = Dataset::create(['name' => 'Farm Survey Data', 'parent_id' => null, 'primary_key' => 'id']); + Dataset::create(['name' => 'Crops', 'parent_id' => $farmSurveyDataset->id, 'primary_key' => 'id']); + Dataset::create(['name' => 'Livestock', 'parent_id' => $farmSurveyDataset->id, 'primary_key' => 'id']); Dataset::create(['name' => 'Locations', 'parent_id' => $farmSurveyDataset->id, 'primary_key' => 'id', 'entity_model' => Location::class]); } diff --git a/resources/views/filament/app/clusters/location-levels/resources/farm-resource/pages/import-locations-and-farms.blade.php b/resources/views/filament/app/clusters/location-levels/resources/farm-entity-resource/pages/import-locations-and-farm-entities.blade.php similarity index 100% rename from resources/views/filament/app/clusters/location-levels/resources/farm-resource/pages/import-locations-and-farms.blade.php rename to resources/views/filament/app/clusters/location-levels/resources/farm-entity-resource/pages/import-locations-and-farm-entities.blade.php diff --git a/resources/views/filament/app/pages/survey-locations/survey-locations-index.blade.php b/resources/views/filament/app/pages/survey-locations/survey-locations-index.blade.php index 3fb6fa72..4a36dd3e 100644 --- a/resources/views/filament/app/pages/survey-locations/survey-locations-index.blade.php +++ b/resources/views/filament/app/pages/survey-locations/survey-locations-index.blade.php @@ -3,9 +3,6 @@ use App\Filament\App\Clusters\LocationLevels\Resources\FarmEntityResource; use App\Filament\App\Pages\SurveyDashboard; - // Points at the new ODK-Entities-backed Farms page - the original FarmResource is - // kept around (unlinked) for comparison during the migration. See - // docs/plans/odk-entities-farm-crud.md. $farmUrl = FarmEntityResource::getUrl(); $surveyDashboardUrl = SurveyDashboard::getUrl(); diff --git a/resources/views/filament/app/resources/farm-resource/widgets/farm-list-header-widget.blade.php b/resources/views/filament/app/resources/farm-entity-resource/widgets/farm-list-header-widget.blade.php similarity index 100% rename from resources/views/filament/app/resources/farm-resource/widgets/farm-list-header-widget.blade.php rename to resources/views/filament/app/resources/farm-entity-resource/widgets/farm-list-header-widget.blade.php diff --git a/resources/views/livewire/data-collection/data-collection-by-farm.blade.php b/resources/views/livewire/data-collection/data-collection-by-farm.blade.php deleted file mode 100644 index 9cf15d2d..00000000 --- a/resources/views/livewire/data-collection/data-collection-by-farm.blade.php +++ /dev/null @@ -1,5 +0,0 @@ -
- @if($visible) - {{ $this->table }} - @endif -
diff --git a/tests/Feature/Crud/AppPanelCrudTest.php b/tests/Feature/Crud/AppPanelCrudTest.php index 93f4782d..cbc2c620 100644 --- a/tests/Feature/Crud/AppPanelCrudTest.php +++ b/tests/Feature/Crud/AppPanelCrudTest.php @@ -1,12 +1,12 @@ Http::response(['token' => 'fake-token'], 200), + // ListFarmEntities::mount() self-heals the local OdkDataset row from Central; + // a 404 short-circuits it to an empty live feed without needing an OData fake. + '*/datasets/Farm_Summary' => Http::response([], 404), + '*/entities/*' => Http::response([], 200), + ]); + $this->team = Team::factory()->create(); + OdkProject::create(['id' => 1, 'owner_type' => Team::class, 'owner_id' => $this->team->id, 'name' => 'Project 1']); + $this->user = createAppUser($this->team); $this->actingAs($this->user); }); test('farm list page loads', function () { - $this->get("/app/{$this->team->id}/location-levels/farms")->assertOk(); + $this->get("/app/{$this->team->id}/location-levels/farm-entities")->assertOk(); }); - test('can bulk delete farm', function () { + test('can delete farm entity via table action', function () { withAppTenant($this->team); $level = new LocationLevel; $level->name = 'Village'; $level->owner_id = $this->team->id; + $level->has_farms = true; $level->save(); $location = Location::create([ @@ -140,16 +151,21 @@ 'code' => 'tv1', ]); - $farm = Farm::create([ + $farmEntity = FarmEntity::create([ 'owner_id' => $this->team->id, 'location_id' => $location->id, 'team_code' => 'farm-001', + 'odk_uuid' => 'uuid-farm-001', + 'odk_version' => 1, ]); - livewire(ListFarms::class) - ->callTableBulkAction(DeleteBulkAction::class, [$farm]); + livewire(ListFarmEntities::class) + ->callTableAction(DeleteAction::class, $farmEntity); + + $this->assertSoftDeleted('farm_entities', ['id' => $farmEntity->id]); - $this->assertDatabaseMissing('farms', ['id' => $farm->id]); + Http::assertSent(fn ($request) => $request->method() === 'DELETE' + && str_contains($request->url(), 'datasets/Farm_Summary/entities/uuid-farm-001')); }); }); diff --git a/tests/Feature/Exports/FarmSurveyDataExportTest.php b/tests/Feature/Exports/FarmSurveyDataExportTest.php new file mode 100644 index 00000000..a411195f --- /dev/null +++ b/tests/Feature/Exports/FarmSurveyDataExportTest.php @@ -0,0 +1,28 @@ +team = Team::factory()->create(); +}); + +test('submissions export builds for a team with no submissions', function () { + $sheets = (new FarmSurveyDataExport($this->team))->sheets(); + + expect($sheets)->not->toBeEmpty(); + + $raw = Excel::raw(new FarmSurveyDataExport($this->team), ExcelWriter::XLSX); + + expect($raw)->not->toBeEmpty(); +}); diff --git a/tests/Feature/Smoke/AppPanelTest.php b/tests/Feature/Smoke/AppPanelTest.php index 4f1d0024..634ac4a4 100644 --- a/tests/Feature/Smoke/AppPanelTest.php +++ b/tests/Feature/Smoke/AppPanelTest.php @@ -111,9 +111,11 @@ ->assertOk(); }); + // The team has no OdkProject, so ListFarmEntities::mount()'s refreshFromCentral() + // short-circuits to an empty live feed without reaching ODK Central at all. test('farms list loads', function () { $this->actingAs($this->user) - ->get("/app/{$this->team->id}/location-levels/farms") + ->get("/app/{$this->team->id}/location-levels/farm-entities") ->assertOk(); }); From 839d002cbb1d1c6adadcde948e17275f19d7f8f1 Mon Sep 17 00:00:00 2001 From: Dave Mills Date: Fri, 31 Jul 2026 12:27:54 +0100 Subject: [PATCH 2/4] Reclaim the 'farms' URL slug for FarmEntityResource The legacy FarmResource that owned app/{tenant}/location-levels/farms is gone, so the entities-backed resource can take the clean URL back. Class names are unchanged; only the slug and the two tests that hit the route by path move. --- .../Clusters/LocationLevels/Resources/FarmEntityResource.php | 2 +- tests/Feature/Crud/AppPanelCrudTest.php | 2 +- tests/Feature/Smoke/AppPanelTest.php | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource.php b/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource.php index 87b7dfaa..6fb9b5f6 100644 --- a/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource.php +++ b/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource.php @@ -32,7 +32,7 @@ class FarmEntityResource extends Resource { protected static ?string $model = FarmEntity::class; - protected static ?string $slug = 'farm-entities'; + protected static ?string $slug = 'farms'; protected static bool $shouldRegisterNavigation = false; diff --git a/tests/Feature/Crud/AppPanelCrudTest.php b/tests/Feature/Crud/AppPanelCrudTest.php index cbc2c620..fd897517 100644 --- a/tests/Feature/Crud/AppPanelCrudTest.php +++ b/tests/Feature/Crud/AppPanelCrudTest.php @@ -132,7 +132,7 @@ }); test('farm list page loads', function () { - $this->get("/app/{$this->team->id}/location-levels/farm-entities")->assertOk(); + $this->get("/app/{$this->team->id}/location-levels/farms")->assertOk(); }); test('can delete farm entity via table action', function () { diff --git a/tests/Feature/Smoke/AppPanelTest.php b/tests/Feature/Smoke/AppPanelTest.php index 634ac4a4..b6a4a27a 100644 --- a/tests/Feature/Smoke/AppPanelTest.php +++ b/tests/Feature/Smoke/AppPanelTest.php @@ -115,7 +115,7 @@ // short-circuits to an empty live feed without reaching ODK Central at all. test('farms list loads', function () { $this->actingAs($this->user) - ->get("/app/{$this->team->id}/location-levels/farm-entities") + ->get("/app/{$this->team->id}/location-levels/farms") ->assertOk(); }); From 8edc48bc417355f11443597569cb4b5fe99d51a2 Mon Sep 17 00:00:00 2001 From: Dave Mills Date: Fri, 31 Jul 2026 12:33:45 +0100 Subject: [PATCH 3/4] Chain the farm import after the location import in the combined wizard Fixes finding M13. The wizard called Excel::import() twice, and for a ShouldQueue + WithChunkReading importer each call builds its own QueueImport -> ReadChunk... -> AfterImportJob chain. Two independent chains on one queue means ordering is whatever the worker pool decides, so with more than one worker the farm chunks could run before the location chunks had committed - and FarmEntityImport::rules() validates its location column with Rule::exists('locations', 'code'). Bus::chain() would not fix it: a chained job calling Excel::import() returns as soon as the chunks are queued. The dependency has to live inside the chain maatwebsite/excel already built, so the farm import is appended to its tail via PendingDispatch -> Queueable::appendToChain(). - new QueueFarmEntityImport job; resolves the media path from the Import id at run time, and its failed() surfaces a start-up failure on the imports table instead of leaving the record silently empty - the farm job's payload drops $data['level'] (a LocationLevel model, which SerializesModels cannot reduce inside an array property) - LocationImport now writes a 'skipped because the location import failed' error onto the dependent farm Import record when dependent_import_id is present; the key is optional for standalone use --- .../Pages/ImportLocationsAndFarmEntities.php | 27 ++- app/Imports/LocationImport.php | 32 +++ app/Jobs/QueueFarmEntityImport.php | 56 +++++ ...rtLocationsAndFarmEntitiesChainingTest.php | 211 ++++++++++++++++++ 4 files changed, 320 insertions(+), 6 deletions(-) create mode 100644 app/Jobs/QueueFarmEntityImport.php create mode 100644 tests/Feature/Imports/ImportLocationsAndFarmEntitiesChainingTest.php diff --git a/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource/Pages/ImportLocationsAndFarmEntities.php b/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource/Pages/ImportLocationsAndFarmEntities.php index 47674362..91e5b7b2 100644 --- a/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource/Pages/ImportLocationsAndFarmEntities.php +++ b/app/Filament/App/Clusters/LocationLevels/Resources/FarmEntityResource/Pages/ImportLocationsAndFarmEntities.php @@ -3,8 +3,8 @@ namespace App\Filament\App\Clusters\LocationLevels\Resources\FarmEntityResource\Pages; use App\Filament\App\Clusters\LocationLevels\Resources\FarmEntityResource; -use App\Imports\FarmEntityImport; use App\Imports\LocationImport; +use App\Jobs\QueueFarmEntityImport; use App\Models\Import; use App\Models\SampleFrame\FarmEntity; use App\Models\SampleFrame\Location; @@ -87,9 +87,6 @@ public function save(): void ]); $locationImport->addMedia(Storage::path($data['upload']))->toMediaCollection(); - $data['import_id'] = $locationImport->id; - - Excel::import(new LocationImport($data), $locationImport->getFirstMediaPath()); // import farms as ODK Central entities $farmImport = Import::create([ @@ -98,9 +95,27 @@ public function save(): void ]); $farmImport->addMedia(Storage::path($data['upload']).'_duplicate')->toMediaCollection(); - $data['import_id'] = $farmImport->id; - Excel::import(new FarmEntityImport($data), $farmImport->getFirstMediaPath()); + // $data['level'] holds a whole LocationLevel model, and SerializesModels does not + // reduce models nested inside an array property - FarmEntityImport never reads it. + $farmData = $data; + unset($farmData['level']); + $farmData['import_id'] = $farmImport->id; + + $data['import_id'] = $locationImport->id; + $data['dependent_import_id'] = $farmImport->id; + + // Farm rows validate against locations this import is still creating + // (FarmEntityImport::rules() -> Rule::exists('locations', 'code')), so the farm import + // must not run concurrently. Appending to the location import's own chain - rather + // than dispatching a second chain - is what actually orders them: a sibling + // Bus::chain link would fire as soon as the location chunks had been *queued*, not + // once they had run. queueImport() is Excel::import() narrowed to the ShouldQueue + // case, so it always hands back the PendingDispatch wrapping that chain, and + // appendToChain() must happen before it falls out of scope and dispatches. + Excel::queueImport(new LocationImport($data), $locationImport->getFirstMediaPath()) + // @phpstan-ignore-next-line PendingDispatch::__call() forwards to Queueable::appendToChain() on the underlying QueueImport job + ->appendToChain(new QueueFarmEntityImport($farmData, $farmImport->id)); Notification::make() ->title(t('Locations and farms are being imported.')) diff --git a/app/Imports/LocationImport.php b/app/Imports/LocationImport.php index 94ebac1b..ddb0e16b 100644 --- a/app/Imports/LocationImport.php +++ b/app/Imports/LocationImport.php @@ -140,6 +140,36 @@ public function chunkSize(): int return 1000; } + /** + * When this import is the first half of the combined locations+farms wizard, the farm + * import is appended to this import's own job chain (see + * ImportLocationsAndFarmEntities::save()), so a failure here aborts the chain and the + * farm import never runs. Without this its Import record would sit empty, reading as + * "nothing happened" rather than "skipped". The key is absent when LocationImport is + * used standalone from LocationLevelResource\Pages\ViewLocationLevel. + */ + protected function failDependentImport(ImportFailed $event): void + { + $dependentImportId = $this->data['dependent_import_id'] ?? null; + + if ($dependentImportId === null) { + return; + } + + Import::find($dependentImportId)?->update([ + 'errors' => [ + [ + 'row' => null, + 'attribute' => null, + 'errors' => [ + 'The farm import was skipped because the location import it depends on failed: ' + .$event->getException()->getMessage(), + ], + ], + ], + ]); + } + public function registerEvents(): array { return [ @@ -148,6 +178,8 @@ public function registerEvents(): array ->update([ 'errors' => $event->getException()->getMessage(), ]); + + $this->failDependentImport($event); }, AfterImport::class => function (AfterImport $event) { Notification::make() diff --git a/app/Jobs/QueueFarmEntityImport.php b/app/Jobs/QueueFarmEntityImport.php new file mode 100644 index 00000000..26dec81b --- /dev/null +++ b/app/Jobs/QueueFarmEntityImport.php @@ -0,0 +1,56 @@ + $data */ + public function __construct(public array $data, public int $importId) {} + + public function handle(): void + { + $import = Import::findOrFail($this->importId); + + Excel::import(new FarmEntityImport($this->data), $import->getFirstMediaPath()); + } + + /** + * Only fires if the farm import could not even be started - once Excel::import() has + * queued its own chunk jobs, FarmEntityImport's own ImportFailed handler owns any + * later failure. Written in the same shape so the imports table renders it the same way. + */ + public function failed(Throwable $exception): void + { + Import::find($this->importId)?->update([ + 'errors' => [ + [ + 'row' => null, + 'attribute' => null, + 'errors' => [$exception->getMessage()], + ], + ], + ]); + } +} diff --git a/tests/Feature/Imports/ImportLocationsAndFarmEntitiesChainingTest.php b/tests/Feature/Imports/ImportLocationsAndFarmEntitiesChainingTest.php new file mode 100644 index 00000000..81f788cf --- /dev/null +++ b/tests/Feature/Imports/ImportLocationsAndFarmEntitiesChainingTest.php @@ -0,0 +1,211 @@ +fillForm([ + 'upload' => ['fake-upload-uuid' => $upload], + 'header_columns' => [ + 'village_code' => 'village_code', + 'village_name' => 'village_name', + 'farm_code' => 'farm_code', + 'family_name' => 'family_name', + ], + 'code_column' => 'village_code', + 'name_column' => 'village_name', + 'level' => $level, + 'user_id' => $user->id, + 'owner_id' => $team->id, + 'location_level_id' => $level->id, + 'location_code_column' => 'village_code', + 'farm_code_column' => 'farm_code', + 'farm_identifiers' => ['family_name'], + 'farm_properties' => [], + ]) + ->call('save') + ->assertHasNoFormErrors(); +} + +describe('ImportLocationsAndFarmEntities chains the farm import after the location import', function () { + + beforeEach(function () { + Http::fake(); + Storage::fake(config('filesystems.default')); + + $this->team = Team::factory()->create(); + $this->user = createAppUser($this->team); + $this->actingAs($this->user); + + $this->level = LocationLevel::create([ + 'owner_id' => $this->team->id, + 'name' => 'Village', + 'has_farms' => true, + ]); + + withAppTenant($this->team); + }); + + test('one queued chain is pushed, with the farm import as its tail', function () { + submitCombinedImportWizard($this->team, $this->user, $this->level); + + Queue::assertPushed(QueueImport::class, 1); + + Queue::assertPushed(QueueImport::class, function (QueueImport $job) { + $tail = unserialize(end($job->chained)); + + return $tail instanceof QueueFarmEntityImport; + }); + }); + + test('the chained farm job carries the farm Import id and no LocationLevel model', function () { + submitCombinedImportWizard($this->team, $this->user, $this->level); + + $farmImport = Import::where('model_type', FarmEntity::class)->sole(); + + Queue::assertPushed(QueueImport::class, function (QueueImport $job) use ($farmImport) { + /** @var QueueFarmEntityImport $tail */ + $tail = unserialize(end($job->chained)); + + return $tail->importId === $farmImport->id + && $tail->data['import_id'] === $farmImport->id + && ! array_key_exists('level', $tail->data); + }); + }); + +}); + +describe('QueueFarmEntityImport', function () { + + beforeEach(function () { + Http::fake(); + Storage::fake(config('filesystems.default')); + + $this->team = Team::factory()->create(); + }); + + test('handle() imports the farm Import record media with a FarmEntityImport', function () { + $upload = combinedImportSpreadsheet(); + + $import = Import::create(['team_id' => $this->team->id, 'model_type' => FarmEntity::class]); + $import->addMedia(Storage::path($upload))->preservingOriginal()->toMediaCollection(); + + Excel::fake(); + + (new QueueFarmEntityImport(['owner_id' => $this->team->id], $import->id))->handle(); + + Excel::assertImported($import->getFirstMediaPath(), fn ($import) => $import instanceof FarmEntityImport); + }); + + test('failed() records the exception on the farm Import record', function () { + $import = Import::create(['team_id' => $this->team->id, 'model_type' => FarmEntity::class]); + + (new QueueFarmEntityImport([], $import->id))->failed(new RuntimeException('queue exploded')); + + expect($import->fresh()->errors->first()['errors'])->toBe(['queue exploded']); + }); + +}); + +describe('LocationImport failure propagation', function () { + + beforeEach(function () { + Http::fake(); + $this->team = Team::factory()->create(); + $this->user = User::factory()->create(); + + $this->locationImportRecord = Import::create(['team_id' => $this->team->id, 'model_type' => 'Location']); + }); + + test('a failed location import explains the skip on the dependent farm Import record', function () { + $farmImportRecord = Import::create(['team_id' => $this->team->id, 'model_type' => FarmEntity::class]); + + $import = new LocationImport(locationImportData($this->team, $this->user, [ + 'import_id' => $this->locationImportRecord->id, + 'dependent_import_id' => $farmImportRecord->id, + ])); + + $handler = $import->registerEvents()[ImportFailed::class]; + $handler(new ImportFailed(new RuntimeException('bad location file'))); + + expect($this->locationImportRecord->fresh()->errors->first())->toBe('bad location file'); + expect($farmImportRecord->fresh()->errors->first()['errors'][0]) + ->toContain('skipped because the location import it depends on failed') + ->toContain('bad location file'); + }); + + test('a standalone location import failure works without a dependent import', function () { + $import = new LocationImport(locationImportData($this->team, $this->user, [ + 'import_id' => $this->locationImportRecord->id, + ])); + + $handler = $import->registerEvents()[ImportFailed::class]; + $handler(new ImportFailed(new RuntimeException('bad location file'))); + + expect($this->locationImportRecord->fresh()->errors->first())->toBe('bad location file'); + }); + +}); + +function locationImportData(Team $team, User $user, array $overrides = []): array +{ + return array_merge([ + 'header_columns' => [ + 'village_code' => 'village_code', + 'village_name' => 'village_name', + ], + 'code_column' => 'village_code', + 'name_column' => 'village_name', + 'owner_id' => $team->id, + 'user_id' => $user->id, + ], $overrides); +} From 88260efa3bf080203654c92160097c8dd5e51e34 Mon Sep 17 00:00:00 2001 From: Dave Mills Date: Fri, 31 Jul 2026 12:35:06 +0100 Subject: [PATCH 4/4] Add change log for the farm CRUD cutover and import chaining --- .../farm-crud-cutover-and-import-chaining.md | 126 ++++++++++++++++++ .../farm-crud-cutover-and-import-chaining.md | 2 +- 2 files changed, 127 insertions(+), 1 deletion(-) create mode 100644 docs/change-logs/farm-crud-cutover-and-import-chaining.md diff --git a/docs/change-logs/farm-crud-cutover-and-import-chaining.md b/docs/change-logs/farm-crud-cutover-and-import-chaining.md new file mode 100644 index 00000000..ec5f65b7 --- /dev/null +++ b/docs/change-logs/farm-crud-cutover-and-import-chaining.md @@ -0,0 +1,126 @@ +# Change log: Farm CRUD cutover + combined-import chaining + +Implements [docs/plans/farm-crud-cutover-and-import-chaining.md](../plans/farm-crud-cutover-and-import-chaining.md), which closes Phase 6 of [odk-entities-farm-crud.md](../plans/odk-entities-farm-crud.md) and finding M13 of [2026-07-19-map-location-to-entity-list-attribute.md](../code-reviews/2026-07-19-map-location-to-entity-list-attribute.md). + +Landed as three commits on `remove-extra-holpa-items`: + +| Commit | Scope | +| --- | --- | +| `e6bd0a7` | Part A — remove the legacy Farm model, `FarmResource` and the HOLPA survey-data models | +| `d249bff` | Part A8 — reclaim the `farms` URL slug for `FarmEntityResource` | +| `573a267` | Part B — chain `FarmEntityImport` after `LocationImport` | + +--- + +## Part A — legacy farm and HOLPA survey-data removal + +### Moved (shared assets, not legacy) + +- `FarmResource/Widgets/FarmListHeaderWidget.php` → `FarmEntityResource/Widgets/FarmListHeaderWidget.php` (namespace updated; `ListFarmEntities` import updated). +- Its Blade view moved too, `filament/app/resources/farm-resource/widgets/` → `.../farm-entity-resource/widgets/`, so the widget's `$view` no longer points into a `farm-resource` tree that has no resource behind it. +- `.../farm-resource/pages/import-locations-and-farms.blade.php` → `.../farm-entity-resource/pages/import-locations-and-farm-entities.blade.php`; `ImportLocationsAndFarmEntities::$view` updated and both empty `farm-resource/` view directories removed. + +### Deleted + +- `FarmResource.php`, `FarmResource/Pages/ListFarms.php`, `FarmResource/Pages/ImportLocationsAndFarms.php` and the whole `FarmResource/` directory. +- `app/Models/SampleFrame/Farm.php`, `app/Imports/FarmImport.php`, `app/Events/FarmImportCompleted.php`, `app/Policies/FarmPolicy.php`. +- `app/Livewire/DataCollection/DataCollectionByFarm.php` + its Blade view (zero references anywhere). +- `HelperService::findFarmLocationDetails()` (no callers) and its now-unused `Farm` / `Str` imports. +- The whole `app/Models/SurveyData/` namespace — `FarmSurveyData`, `Crop`, `Livestock` — plus `app/Models/Interfaces/RepeatModel.php` and both now-empty directories. + +### Edited + +| File | Change | +| --- | --- | +| `app/Providers/AppServiceProvider.php` | dropped `Gate::policy(Farm::class, FarmPolicy::class)` and both imports | +| `app/Models/Team.php` | removed `farms()`; `pilotProgress` now `$this->xlsforms()->whereHas('xlsformVersions.submissions')->exists()`; `dataCollectionProgress` the same with `->where('test_data', false)` | +| `app/Models/SampleFrame/Location.php` | removed `farms()` and the three farm-completion attributes; `farmEntities()` and `farms_all_count` kept | +| `app/Livewire/DataCollection/DataCollectionByLocation.php` | the "Farm Counts" `ColumnGroup` collapsed to a single `farms_all_count` "Farms" column | +| `app/Filament/Admin/Widgets/DataCollectedWidget.php` | removed the "Farms surveyed" stat | +| `app/Exports/DataExport/DatasetExport.php` | removed the `Farm` import and `@var` docblock; runtime behaviour untouched | +| `app/Filament/Tables/Actions/ImportFarmsAction.php` | `model_type => FarmEntity::class` (was `Farm::class`) — closes the cosmetic gap recorded in the earlier plan | +| `database/seeders/Prep/DatasetSeeder.php` | the `Farms` row deleted; `Farm Survey Data` / `Crops` / `Livestock` rows kept but their `entity_model` keys dropped | +| `CLAUDE.md` | `Models/SampleFrame/` now lists `FarmEntity`; the `Models/SurveyData/` line removed | + +Comments across `FarmEntityResource`, `ListFarmEntities`, `ImportLocationsAndFarmEntities`, `FarmEntityImport`, `FarmEntityPolicy`, `LocationLevelResource`, `OdkFarmEntityService` and `survey-locations-index.blade.php` that described coexistence with the legacy resource, or that named the deleted classes, were rewritten or dropped. + +### Behaviour change worth knowing + +`Team::pilotProgress` previously meant "a farm has at least one submission"; it now means "the team has at least one submission". Nothing has linked submissions to farms since HOLPA submission processing was removed in `4a93d71`, so the old expression could only ever return `false` — the new one is strictly closer to what the dashboard tile is trying to say. The same reasoning applies to `dataCollectionProgress`, which read `farms.household_form_completed` / `fieldwork_form_completed`, columns nothing has written for just as long. + +### Schema + +- Deleted the four `create` migrations (`create_farms_table`, `create_farm_survey_data_table`, `create_crops_table`, `create_livestocks_table`) so fresh installs never build the tables. +- Added `2026_07_31_000001_drop_legacy_farm_tables.php`, which `dropIfExists()`es `crops`, `livestocks`, `farm_survey_data` and `farms` so existing dev databases converge. A no-op on a fresh install. No FK constraints existed between any of them. +- No column-level surgery was needed: `farm_survey_data.farm_id` disappeared with its table, and nothing outside the four tables pointed at them. +- **Note for later:** existing dev databases still hold seeded `Dataset` rows whose `entity_model` column names the now-deleted classes. Nothing reads `entity_model` (the only readers — `Dataset::globalEntries` / `teamEntries` / `databaseTable` — have no consumers), so no data migration was written. Those strings are stale labels, not live pointers. + +### What deliberately survived + +`App\Exports\DataExport\FarmSurveyDataExport`, `FarmSurveyDatasetExport` and `DatasetExport` share names with the deleted models but are driven entirely by `Dataset` rows looked up **by name** plus generic `Entity`/`EntityValue` rows written by `OdkSubmissionService`. They never referenced the model classes. That is why the `Farm Survey Data`, `Crops` and `Livestock` seeder rows had to stay — `FarmSurveyDataExport` `abort(500)`s without the parent row. + +### A8 — the `farms` slug + +With the old resource gone, `FarmEntityResource::$slug` went from `farm-entities` back to `farms`, restoring `app/{tenant}/location-levels/farms`. Class names unchanged. `survey-locations-index.blade.php` already resolves the URL through `FarmEntityResource::getUrl()` and needed no edit; only the two tests that hit the route by path moved. Kept as its own commit so it can be reverted independently of the deletions. + +--- + +## Part B — chaining the combined import + +### Root cause + +`ImportLocationsAndFarmEntities::save()` called `Excel::import()` twice and discarded both return values. For a `ShouldQueue` + `WithChunkReading` importer, `ChunkReader::read()` builds `(new QueueImport($import))->chain([...ReadChunk jobs, AfterImportJob])` and returns it inside a `PendingDispatch` that dispatches on destruction — so the page produced **two independent chains on one queue**. Ordering was then whatever the worker pool decided, and `FarmEntityImport::rules()` validates its location column with `Rule::exists('locations', 'code')`, so with more than one worker the farm import could fail wholesale with per-row "location does not exist" errors. It only worked because a single FIFO worker happened to preserve dispatch order. + +`Bus::chain([$locationImport, $farmImport])` would not have fixed it: a chained job that itself calls `Excel::import()` returns as soon as its chunks are *queued*, so the next outer link starts immediately. + +### `app/Jobs/QueueFarmEntityImport.php` (new) + +A plain `ShouldQueue` job holding `array $data` and `int $importId`. `handle()` resolves the `Import` record and runs `Excel::import(new FarmEntityImport($this->data), $import->getFirstMediaPath())` — the media path is resolved at run time rather than serialised as an absolute path built at request time. `failed()` writes the exception onto the `Import` record in the same shape `FarmEntityImport`'s own `ImportFailed` handler uses, so a failure to even *start* the farm import shows on the imports table instead of being silent. + +### `ImportLocationsAndFarmEntities::save()` + +```php +Excel::queueImport(new LocationImport($data), $locationImport->getFirstMediaPath()) + ->appendToChain(new QueueFarmEntityImport($farmData, $farmImport->id)); +``` + +`Queueable::appendToChain()` adds to the chain `ChunkReader` already built, after its trailing `AfterImportJob` — `Queueable::chain()` would have *replaced* `$chained` and thrown away every `ReadChunk` job. `PendingDispatch::__call()` forwards the call to the underlying `QueueImport`. + +The plan called for `Excel::import()` behind an `instanceof PendingDispatch` guard to satisfy phpstan. `Excel::queueImport()` was used instead: it is the same call narrowed to the `ShouldQueue` case and is annotated `PendingDispatch` on the facade, so the guard — which was never a real branch — is gone. One `@phpstan-ignore-next-line` remains on `appendToChain()`, which phpstan cannot see through `PendingDispatch::__call()`. + +`$farmData` is a copy of `$data` with `$data['level']` unset: that key holds a whole `LocationLevel` **model**, and `SerializesModels` does not reduce models nested inside array properties. `FarmEntityImport` never reads it. + +On `QUEUE_CONNECTION=sync` the chain runs inline in order, so local and test behaviour is unchanged apart from the ordering guarantee. + +### `LocationImport` failure propagation + +If the location import fails the chain aborts and `QueueFarmEntityImport` never runs, leaving the farm `Import` row empty — reading as "nothing happened" rather than "skipped". `save()` now also passes `$data['dependent_import_id']`, and `LocationImport`'s `ImportFailed` handler calls a new `failDependentImport()` that writes an explanatory error onto that record. The key is optional: `LocationImport` is still used standalone from `LocationLevelResource\Pages\ViewLocationLevel`. + +--- + +## Tests + +| File | Change | +| --- | --- | +| `tests/Feature/Crud/AppPanelCrudTest.php` | the `App panel CRUD — Farm` block replaced by `App panel CRUD — FarmEntity`: the list page loads, and `DeleteAction` soft-deletes the row *and* issues the `DELETE .../datasets/Farm_Summary/entities/{uuid}` call to Central. The delete path had no coverage before. | +| `tests/Feature/Smoke/AppPanelTest.php` | "farms list loads" repointed at the new resource | +| `tests/Feature/Exports/FarmSurveyDataExportTest.php` (new) | pins the plan's claim that the submissions export is name-driven, not model-driven: it must still build after the `SurveyData` models and their `entity_model` pointers were removed | +| `tests/Feature/Imports/ImportLocationsAndFarmEntitiesChainingTest.php` (new) | the M13 regression test — submitting the wizard under `Queue::fake()` pushes exactly **one** `QueueImport` whose `chained` tail deserialises to a `QueueFarmEntityImport` carrying the farm `Import` id and no `level` key. Plus `QueueFarmEntityImport::handle()`/`failed()`, and `LocationImport`'s failure path with and without `dependent_import_id`. | + +The smoke test needed no `Http::fake()` treatment beyond what was already there: a team with no `OdkProject` makes `refreshFromCentral()` short-circuit before reaching Central. (The plan predicted this would come from `resolveEntityListName()` returning `null`; that method is now hardcoded to `'Farm_Summary'`, so the `findOdkDataset()` null-project branch is what does it.) + +## Verification + +- `./vendor/bin/pint` — clean. +- `./vendor/bin/phpstan analyse` — 164 errors, down from a 346-error baseline on the branch point. Nothing new in the touched application files; the 10 in the new test file are the usual Pest `$this->` property noise this suite already carries. +- `./vendor/bin/pest` — 167 passed, 377 assertions. +- The plan's verification grep returns only the intended survivors: the three `FarmSurveyData*Export` classes, their two call sites (`XlsformsTableView`, `ExportDataAction`), the seeder's `Dataset` names and the new export test. + +## Still open + +Manual verification from the plan has **not** been done and is left for whoever runs the branch: + +1. Part A — app-panel survey dashboard (both progress tiles), Survey Locations → "List of farms", the location-levels "# of Farms" column, the admin dashboard without the "Farms surveyed" stat, and the submissions export download from the xlsforms table. +2. Part B — a real ODK Central project with **at least two queue workers**, importing a combined spreadsheet whose farms reference locations that do not yet exist locally; then the negative case, where the location half fails validation and the farm `Import` record should show the "skipped" error. + +Follow-ups carried forward from the plan are unchanged: location-deletion behaviour after the `nullOnDelete` change was never manually retested; submission → farm linking is still unimplemented (and should be built on `Dataset`/`Entity`/`EntityValue`, not new per-repeat-group tables); Create/Edit form fields still don't match the Farm Registration XLSForm; entities deleted directly on Central are not mirrored locally; a cleared entity property reappears in the Edit form as a blank row. diff --git a/docs/plans/farm-crud-cutover-and-import-chaining.md b/docs/plans/farm-crud-cutover-and-import-chaining.md index 30e7cff3..460480ec 100644 --- a/docs/plans/farm-crud-cutover-and-import-chaining.md +++ b/docs/plans/farm-crud-cutover-and-import-chaining.md @@ -1,6 +1,6 @@ # Plan: Finish the ODK-Entities farm CRUD cutover (remove legacy Farm + HOLPA survey-data models) + chain the combined import -**Status: Not Started** +**Status: Completed** — see [docs/change-logs/farm-crud-cutover-and-import-chaining.md](../change-logs/farm-crud-cutover-and-import-chaining.md). All steps A1–A8 and B1–B4 landed in three commits on `remove-extra-holpa-items`. Two deviations from the plan as written, both recorded in the change log: A1 also moved the `FarmListHeaderWidget` Blade view (a third shared asset the plan didn't list), and B2 uses `Excel::queueImport()` rather than `Excel::import()` behind an `instanceof PendingDispatch` guard — same call, correctly typed, so the guard was unnecessary. The manual verification steps (5 and 6, which need a real ODK Central project and two queue workers) have not been performed. Wraps up [odk-entities-farm-crud.md](odk-entities-farm-crud.md) — its Phase 6 (cutover) and the one known open finding from [2026-07-19-map-location-to-entity-list-attribute.md](../code-reviews/2026-07-19-map-location-to-entity-list-attribute.md) (M13). Two independent parts; either can land first, but Part A deletes `FarmResource\Pages\ImportLocationsAndFarms`, so doing Part A first avoids fixing the concurrency bug in a file that is about to be deleted.