overleaf/services/web/migrations
Eric Mc Sween 65313e6692 Merge pull request #18044 from overleaf/em-fix-migration-order
Fix doc versions migration order

GitOrigin-RevId: bad0297cafcc29c0890dbc0d6f5dcc159ded838b
2024-04-26 08:04:18 +00:00
..
lib Merge pull request #15748 from overleaf/em-drop-docops-collection 2024-02-01 09:03:46 +00:00
20190720165251_create_migrations.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190730093801_script_example.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145001_create_contacts_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145002_create_deletedProjects_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145003_create_deletedSubscriptions_indexes.js Merge pull request #4806 from overleaf/jk-un-tag-migration 2021-08-20 08:03:31 +00:00
20190912145004_create_docHistoryIndex_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145005_create_docHistory_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145006_create_docOps_indexes.js Merge pull request #15748 from overleaf/em-drop-docops-collection 2024-02-01 09:03:46 +00:00
20190912145007_create_docSnapshots_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145008_create_docs_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145009_create_githubSyncEntityVersions_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145010_create_githubSyncProjectStates_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145011_create_githubSyncUserCredentials_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145012_create_institutions_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145013_create_messages_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145014_create_notifications_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145015_create_oauthAccessTokens_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145016_create_oauthApplications_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145017_create_oauthAuthorizationCodes_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145018_create_projectHistoryFailures_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145019_create_projectHistoryLabels_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145020_create_projectHistoryMetaData_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145021_create_projectHistorySyncState_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145022_create_projectImportFailures_indexes.js Merge pull request #6247 from overleaf/jpa-cleanup-mongo 2022-01-11 09:03:34 +00:00
20190912145023_create_projectInvites_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145024_create_projects_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145025_create_publishers_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145026_create_rooms_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145027_create_spellingPreferences_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145028_create_subscriptions_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145029_create_tags_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145030_create_templates_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145031_create_tokens_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20190912145032_create_users_indexes.js [web] Upgrade Prettier to match version in monorepo root (#6231) 2022-01-11 09:03:23 +00:00
20190912145033_create_userstubs_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20191106102104_saml-log-indexes.js Merge pull request #6247 from overleaf/jpa-cleanup-mongo 2022-01-11 09:03:34 +00:00
20191107191318_saml-indentifiers-index.js Merge pull request #6247 from overleaf/jpa-cleanup-mongo 2022-01-11 09:03:34 +00:00
20200110183327_brandVarationIdIndex.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20200120163346_atlas_recommended_indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20200210084301_remove-duplicate-deleted-things.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20200210121103_uniqueify-deletedthings-indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20200302143624_users_affiliationUnchecked_index.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20200522145727_dropProjectImportFailures.js Merge pull request #6247 from overleaf/jpa-cleanup-mongo 2022-01-11 09:03:34 +00:00
20200522145741_dropProjectImportBatchRecords.js Merge pull request #6247 from overleaf/jpa-cleanup-mongo 2022-01-11 09:03:34 +00:00
20200608213302_saml-cache-indexes.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20200729120824_update_subscriptions_manager_ids_index.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20201106094956_active-projects-index-with-id.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20210310111225_create_deletedFiles_projectId_index.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20210407085118_token-expiry-with-ttl-index.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20210408123210_create_docs_project_id_deleted_deletedAt_index.js Merge pull request #4303 from overleaf/sk-tag-saas-migrations 2021-07-28 02:05:49 +00:00
20210721081758_create_history_display_index.js Create migration to add history display state index (#4335) 2021-07-28 02:06:20 +00:00
20210726083523_convert_confirmedAt_strings_to_dates.js Merge pull request #4350 from overleaf/ns-migration-tags 2021-08-04 02:05:43 +00:00
20210726083523_convert_split_tests_assigned_at_strings_to_dates.js Merge pull request #5364 from overleaf/ta-split-test-assignment-date-backfill 2021-10-22 08:04:49 +00:00
20210727123346_ce_sp_backfill_deleted_files.js Merge pull request #4352 from overleaf/sk-add-ce-sp-migration-deleted-files 2021-08-04 11:13:55 +00:00
20210727150530_ce_sp_backfill_deleted_docs.js Merge pull request #4352 from overleaf/sk-add-ce-sp-migration-deleted-files 2021-08-04 11:13:55 +00:00
20210728115327_ce_sp_backfill_dummy_doc_meta.js Merge pull request #4352 from overleaf/sk-add-ce-sp-migration-deleted-files 2021-08-04 11:13:55 +00:00
20210924140139_splittests-name-index.js Merge pull request #6247 from overleaf/jpa-cleanup-mongo 2022-01-11 09:03:34 +00:00
20220105123000_cleanup_unused_collections.js Merge pull request #6247 from overleaf/jpa-cleanup-mongo 2022-01-11 09:03:34 +00:00
20220105130000_fix_saml_indexes.js Merge pull request #6297 from overleaf/jpa-fix-index 2022-01-11 09:03:37 +00:00
20220222095146_split_tests_analytics_enabled.js Merge pull request #6785 from overleaf/em-split-tests-analytics-enabled 2022-03-01 09:04:15 +00:00
20220811111800_create_dropboxEntities_index.js Merge pull request #9212 from overleaf/jpa-record-dropbox-file-id 2022-08-12 08:06:04 +00:00
20220815105500_create_dropboxProjects_index.js Merge pull request #9291 from overleaf/jpa-dropbox-project-id 2022-08-18 08:03:38 +00:00
20220817120900_create_dropboxProjects_index.js Merge pull request #9447 from overleaf/jpa-dropbox-refresh-project-folder-names 2022-08-31 08:03:40 +00:00
20220825160708_recreate_dropboxEntities.js Merge pull request #9455 from overleaf/em-recreate-dropbox-entities 2022-09-02 08:05:09 +00:00
20220826104236_disable_alpha_beta_program.js [migration] disable betaProgram in CE/SP (#9418) 2022-09-01 08:03:41 +00:00
20220830140459_create_index_user_labsProgram_labsProgramGalileo.js Merge pull request #9466 from overleaf/jk-add-labs-program-for-galileo 2022-09-22 08:03:59 +00:00
20220913105500_create_auditLog_indexes.js Merge pull request #9617 from overleaf/msm-audit-log-collections 2022-09-30 08:04:17 +00:00
20220913125500_migrate_auditLog_to_collections.js Merge pull request #16433 from overleaf/jpa-migrate-audit-log-dev-env 2024-01-11 09:04:35 +00:00
20220929193200_add_auditLog_indexes.js Merge pull request #9800 from overleaf/msm-audit-indices 2022-09-30 08:05:02 +00:00
20221111111111_ce_sp_convert_archived_state.js Merge pull request #10442 from overleaf/jpa-convert-archived-trashed 2022-11-15 09:07:09 +00:00
20221122191857_project_history_chunks_indexes.js Merge pull request #10644 from overleaf/em-chunk-store-mongo 2022-11-29 09:04:40 +00:00
20230110140452_rename_recurly_cached_status.js Merge pull request #11445 from overleaf/revert-11432-revert-11154-tm-lg-subscription-recurly-cache-name 2023-01-27 09:03:58 +00:00
20230124092607_clear_old_2fa_setup.js Merge pull request #11360 from overleaf/jpa-encrypt-2fa-secret 2023-02-17 09:03:44 +00:00
20230207134844_group_invite_emails_to_lowercase.js Merge pull request #11711 from overleaf/tm-invite-emails-lowercase-readd-migration 2023-02-28 09:04:38 +00:00
20230217205311_fix_deleted_history_chunks_index.js Merge pull request #11890 from overleaf/em-fix-deleted-chunks-indexes 2023-02-23 09:05:50 +00:00
20230315170739_back_fill_doc_rev.js Merge pull request #12254 from overleaf/jpa-back-fill-doc-rev 2023-03-17 09:03:58 +00:00
20230405190240_oauth_refresh_tokens_index.js Merge pull request #12536 from overleaf/em-oauth-refresh-tokens-index 2023-04-11 08:04:06 +00:00
20230406125632_oauth_tokens_ttl.js Merge pull request #12545 from overleaf/em-oauth-tokens-ttl 2023-04-11 08:04:23 +00:00
20230426095212_personal_oauth_tokens_user_index.js [web] Issue Oauth2 access tokens (#12788) 2023-04-28 08:04:10 +00:00
20230428154643_history_chunks_garbage_collection_index.js Merge pull request #12861 from overleaf/em-ds-delete-chunk-migration 2023-05-02 19:45:57 +00:00
20230502180757_server_pro_oauth_indexes.js Merge pull request #12912 from overleaf/em-server-pro-oauth-indexes 2023-05-04 08:05:48 +00:00
20230512100122_ensure_history_migration.js Merge pull request #13082 from overleaf/msm-migration-check-history 2023-05-17 08:05:59 +00:00
20230616153016_user_features_updated_at_index.js Merge pull request #13497 from overleaf/jdt-extract-user-feature-set 2023-07-17 10:43:45 +00:00
20230817081910_back_fill_gitBridge_feature_server_pro.js Merge pull request #14383 from overleaf/jpa-server-pro-feature-refresh-migration 2023-08-18 08:05:53 +00:00
20230928092537_backfill_subscriptions_managed_users_feature_flag.js Merge pull request #15084 from overleaf/ab-managers-users-ga 2023-10-09 08:04:11 +00:00
20231016101457_drop_history_display_index.js Merge pull request #15270 from overleaf/em-drop-history-display-index 2023-10-18 08:04:23 +00:00
20231025094810_sso_config_certificates_array.js Merge pull request #15223 from overleaf/ab-sso-multiple-certificates-ui 2023-10-30 09:04:38 +00:00
20231030160030_managed_users_enabled.js adding DB migration for managedUserEnabled flag (#15516) 2023-11-01 09:02:57 +00:00
20231031164732_drop_redundant_indexes.js Merge pull request #15541 from overleaf/em-drop-redundant-indexes 2023-11-03 09:00:19 +00:00
20231101153447_partial_last_opened_index.js Merge pull request #15555 from overleaf/em-partial-last-opened-index 2023-11-03 09:00:16 +00:00
20231101191643_drop_unused_indexes.js Merge pull request #15562 from overleaf/em-drop-unused-indexes 2023-11-07 09:05:04 +00:00
20231101204352_drop_unused_saas_indexes.js Merge pull request #15562 from overleaf/em-drop-unused-indexes 2023-11-07 09:05:04 +00:00
20231105000000_move_doc_versions_from_docops_to_docs.js Merge pull request #18044 from overleaf/em-fix-migration-order 2024-04-26 08:04:18 +00:00
20231110192205_drop_docops_collection.js Merge pull request #15748 from overleaf/em-drop-docops-collection 2024-02-01 09:03:46 +00:00
20231113173237_tokens_user_id_index.js Merge pull request #15728 from overleaf/td-lg-expire-password-tokens 2023-11-22 09:04:30 +00:00
20240115172206_oauth_issuer_index.js Merge pull request #16527 from overleaf/jpa-wf-hint-oauth 2024-01-17 09:03:06 +00:00
20240220130452_remove_deleted_users_from_token_access_refs.js Merge pull request #17023 from overleaf/ii-token-access-null-values 2024-02-26 09:03:13 +00:00
README.md Merge pull request #15921 from overleaf/jpa-migrations-via-rake 2023-11-24 09:05:00 +00:00

Migrations

Migrations for the app environment live in this folder, and use the East migration framework.

We have a npm script which wraps east: npm run migrations -- ...

For example:

npm run migrations -- list -t 'server-ce'

For SAAS, use the rake tasks for staging/production

rake deploy:migrations:list[staging]

Environments and Tags

Overleaf is deployed in three different environments:

  • server-ce: community edition installations (the base system)
  • server-pro: server pro installations
  • saas: the production overleaf site

All migrations are tagged with the environments they should run in. For example, a migration that should run in every environment would be tagged with ['server-ce', 'server-pro', 'saas'].

When invoking east, we specify the relevant tags with the -t or --tags flag. Our adapter will refuse to run if this flag is not set.

Creating new migrations

To create a new migration, run:

npm run migrations -- create <migration name>

This command will create a new migration file in the migrations folder, based on a template. The template provides migrate and rollback methods, which are run by the east binary when running the migrations. rollback should undo the changes made in migrate.

Running scripts as a migration

To run a script in a migration file, look at migrations/20190730093801_script_example.js, which runs the script scripts/example/script_for_migration.js. This uses a method where the script can be run standalone via node, or through the migrations' mechanism.

Running migrations

To run all migrations in a server-ce environment:

npm run migrations -- migrate -t 'server-ce'
# Note: They are run by default on container start.

To run all migrations in a SAAS environment use the rake task:

# list first and check that only your newly added migration is shown. If not, ask in the dev channel for help.
rake deploy:migrations:list[staging]
# After confirming the listing, run the migrations
rake deploy:migrations[staging]

To run all migrations in the dev-env:

make services/web/migrate
# Note: "make install" will pick them up as well

The -t flag also works with other east commands like rollback, and list.

For other options, or for information on how to roll migrations back, take a look at the East documentation.

Tips

Try to use Mongo directly via the db object instead of using Mongoose models. Migrations will need to run in the future, and model files can change. It's unwise to make the migrations depend on code which might change.

Note: Running east rollback without any arguments rolls back all migrations, which you may well not want.