1040 commits

Author	SHA1	Message	Date
Eric Mc Sween	6269ace987	Decaf cleanup: move functions to top level	2020-05-15 09:59:32 -04:00
Eric Mc Sween	3d000bcbe6	Decaf cleanup: use arrow functions for callbacks	2020-05-15 09:59:32 -04:00
Eric Mc Sween	751d9ea718	Decaf cleanup: simplify loops	2020-05-15 09:59:32 -04:00
Eric Mc Sween	7a5a782dc7	Decaf cleanup: camel case variables	2020-05-15 09:59:26 -04:00
Eric Mc Sween	ceb77f4c93	Decaf cleanup: remove unused variables	2020-05-14 16:54:08 -04:00
Eric Mc Sween	6b5760ca28	Decaf cleanup: simplify null checks	2020-05-14 16:53:22 -04:00
Eric Mc Sween	f2c67b66fa	Decaf cleanup: remove default callbacks	2020-05-14 16:50:04 -04:00
Eric Mc Sween	2bff83137c	Decaf cleanup: unnecessary returns	2020-05-14 16:35:10 -04:00
Eric Mc Sween	819aa378d9	Decaf cleanup: remove Array.from()	2020-05-14 16:32:05 -04:00
Eric Mc Sween	16c0ed23db	Fix tests after decaf cleanup ... * Camel casing in logs * The Express request object always has query and body properties	2020-05-11 11:37:59 -04:00
Eric Mc Sween	0b1c7e90af	Decaf cleanup: remove unused variables	2020-05-11 11:26:28 -04:00
Eric Mc Sween	3acb970442	Decaf cleanup: simplify stubbed class	2020-05-11 11:24:07 -04:00
Eric Mc Sween	526ef25fcf	Decaf cleanup: unnecessary returns	2020-05-11 11:20:55 -04:00
Eric Mc Sween	a2a1914a53	Use max_doc_length setting to limit incoming doc size	2020-05-11 11:15:37 -04:00
Eric Mc Sween	cb959ddfc1	Decaf cleanup: use arrow functions for callbacks	2020-05-11 11:14:37 -04:00
Eric Mc Sween	e4ac63dd19	Decaf cleanup: move functions to top level	2020-05-11 11:12:15 -04:00
Eric Mc Sween	64a881461f	Decaf cleanup: camel case variables	2020-05-11 11:07:15 -04:00
Eric Mc Sween	fc73bbe1a5	Decaf cleanup: simplify null checks	2020-05-11 10:52:06 -04:00
Eric Mc Sween	80ea49c69c	Decaf cleanup: remove __guard__	2020-05-11 10:47:27 -04:00
Eric Mc Sween	814ac40e07	Decaf cleanup: unnecessary returns	2020-05-11 10:45:39 -04:00
Eric Mc Sween	3385ec5f26	Decaf cleanup: unnecessary Array.from()	2020-05-11 10:43:22 -04:00
Eric Mc Sween	41c0899b0c	Add a test for document size slightly over max doc length	2020-05-11 10:41:32 -04:00
Eric Mc Sween	f99125c65a	Decaf cleanup: camel case variables	2020-05-08 15:01:34 -04:00
Eric Mc Sween	75f9b0ff10	Decaf cleanup: handle errors	2020-05-08 15:01:34 -04:00
Eric Mc Sween	150c4a88f1	Decaf cleanup: simplify null checks	2020-05-08 15:01:34 -04:00
Eric Mc Sween	dc5d77998c	Decaf cleanup: remove unnecessary returns	2020-05-08 15:01:34 -04:00
Eric Mc Sween	ff2d31c066	Decaf cleanup: remove Array.from()	2020-05-08 15:01:34 -04:00
Eric Mc Sween	e8f935d046	Make max JSON request size configurable and default to 8 MB ... This is to allow multi-document updates, for example when creating a new project from a zip file.	2020-05-08 15:01:34 -04:00
Eric Mc Sween	e60d9237d0	Decaf cleanup: simplify null checks	2020-05-08 15:01:34 -04:00
Eric Mc Sween	18b92adcef	Decaf cleanup: remove __guard__()	2020-05-08 15:01:34 -04:00
Eric Mc Sween	7b2420413c	Decaf cleanup: unnecessary returns and arrow functions in callbacks	2020-05-08 15:01:33 -04:00
Eric Mc Sween	53d79d86a9	Decaf cleanup: remove Array.from()	2020-05-08 14:54:54 -04:00
Tim Alby	9f6ea07002	fix SyntaxError on export var	2020-05-06 12:17:08 +02:00
Tim Alby	cf87daa754	fix Metrics module stub	2020-05-06 12:17:03 +02:00
Tim Alby	8176cb3e8d	remove requires for coffee-script package in tests	2020-05-06 12:16:59 +02:00
Tim Alby	ed7d277f98	lint & format app.js	2020-05-06 12:16:53 +02:00
Tim Alby	e089cfc93c	format config/settings.defaults.js & lib/diff_match_patch.js	2020-05-06 12:16:48 +02:00
Tim Alby	4184976099	fix lint errors in config/settings.defaults.js	2020-05-06 12:16:10 +02:00
Tim Alby	96e7a668b7	disable linting for lib/diff_match_patch.js	2020-05-06 12:16:04 +02:00
Tim Alby	6fdc094b5c	Uninstall coffee-script	2020-05-06 12:13:24 +02:00
Tim Alby	29d5e8f8b9	prettier: convert individual decaffeinated files to Prettier format	2020-05-06 12:13:11 +02:00
Tim Alby	877bae34b3	decaffeinate: convert individual files to js	2020-05-06 12:13:10 +02:00
Tim Alby	86c6c38a6e	decaffeinate: rename individual coffee files to js files	2020-05-06 12:13:07 +02:00
Tim Alby	0bae747b3e	prettier: convert test/stress decaffeinated files to Prettier format	2020-05-06 12:13:07 +02:00
Tim Alby	02b29bfaa1	decaffeinate: rename test/stress/coffee to test/stress/js	2020-05-06 12:13:03 +02:00
decaffeinate	86a3b0842a	decaffeinate: Run post-processing cleanups on run.coffee	2020-05-06 12:12:57 +02:00
decaffeinate	3b6c0d8ca6	decaffeinate: Convert run.coffee to JS	2020-05-06 12:12:55 +02:00
decaffeinate	b6cc463a1e	decaffeinate: Rename run.coffee from .coffee to .js	2020-05-06 12:12:53 +02:00
Tim Alby	d15738cb98	prettier: convert test/acceptance decaffeinated files to Prettier format	2020-05-06 12:12:47 +02:00
Tim Alby	9680e62bb1	decaffeinate: rename test/acceptance/coffee to test/acceptance/js	2020-05-06 12:12:41 +02:00
decaffeinate	24ac4d4935	decaffeinate: Run post-processing cleanups on ApplyingUpdatesToADocTests.coffee and 14 other files	2020-05-06 12:12:36 +02:00
decaffeinate	adffde3059	decaffeinate: Convert ApplyingUpdatesToADocTests.coffee and 14 other files to JS	2020-05-06 12:12:17 +02:00
decaffeinate	f46fe5be7f	decaffeinate: Rename ApplyingUpdatesToADocTests.coffee and 14 other files from .coffee to .js	2020-05-06 12:12:02 +02:00
Tim Alby	9a2f8ecbd4	prettier: convert test/unit decaffeinated files to Prettier format	2020-05-06 12:11:36 +02:00
Tim Alby	bb20394243	decaffeinate: rename test/unit/coffee to test/unit/js	2020-05-06 12:11:30 +02:00
decaffeinate	21850f1e6f	decaffeinate: Run post-processing cleanups on DiffCodecTests.coffee and 23 other files	2020-05-06 12:11:22 +02:00
decaffeinate	c781526af0	decaffeinate: Convert DiffCodecTests.coffee and 23 other files to JS	2020-05-06 12:10:51 +02:00
decaffeinate	6c4d7fb838	decaffeinate: Rename DiffCodecTests.coffee and 23 other files from .coffee to .js	2020-05-06 12:10:11 +02:00
Tim Alby	dbf9e88dc3	prettier: convert app/js decaffeinated files to Prettier format	2020-05-06 12:09:33 +02:00
Tim Alby	a519980c10	decaffeinate: rename app/coffee dir to app/js	2020-05-06 12:09:23 +02:00
decaffeinate	dad1d1212f	decaffeinate: Run post-processing cleanups on DeleteQueueManager.coffee and 58 other files	2020-05-06 12:09:15 +02:00
decaffeinate	1fa8882674	decaffeinate: Convert DeleteQueueManager.coffee and 58 other files to JS	2020-05-06 12:08:21 +02:00
decaffeinate	249b7247b5	decaffeinate: Rename DeleteQueueManager.coffee and 58 other files from .coffee to .js	2020-05-06 12:07:29 +02:00
Tim Alby	e2e8593f57	decaffeinate: add eslint and prettier packages	2020-05-06 12:06:28 +02:00
Tim Alby	16c98b3d15	decaffeinate: update .gitignore	2020-05-06 12:04:26 +02:00
Tim Alby	569a1f5824	decaffeinate: update build scripts to es	2020-05-06 12:04:19 +02:00
Tim Alby	bb9a5340f4	update build scripts to 2.2.0	2020-05-06 12:03:41 +02:00
Miguel Serrano	93a5ee063a	regenerated lockfile	2020-04-27 13:41:23 +02:00
Miguel Serrano	f935c392bc	Merge branch 'master' into sk-upgrade-dependencies	2020-04-27 13:38:21 +02:00
Brian Gough	79c934759c	add default redis settings for project history	2020-04-22 14:04:28 +01:00
Brian Gough	2e24d1670c	remove old unused ioredis keepalive option	2020-04-22 13:51:14 +01:00
Brian Gough	61da130cf4	keep maxRetriesPerRequest for project_history redis config	2020-04-22 13:50:39 +01:00
Brian Gough	a51f61a555	remove redis migration code	2020-04-21 14:48:47 +01:00
Brian Gough	248edc03fa	add comment about the two history clients	2020-04-21 14:44:19 +01:00
Brian Gough	af93193d6e	remove new_project_history and use project_history instead	2020-04-21 14:43:48 +01:00
Brian Gough	8e210fe441	update unit tests	2020-04-21 14:41:30 +01:00
Brian Gough	ac68f59487	Merge branch 'master' into bg-use-separate-redis-for-project-history	2020-04-16 15:48:21 +01:00
Brian Gough	55cbfd9962	Merge branch 'master' into bg-add-queue-metrics	2020-04-06 12:12:12 +01:00
Brian Gough	c095feaa06	upgrade logger-sharelatex	2020-04-06 10:43:53 +01:00
Brian Gough	beb3691795	add metrics for redis get/update	2020-04-06 10:00:48 +01:00
Brian Gough	2b72ec49a1	add comments for redis metrics	2020-04-02 11:33:52 +01:00
Brian Gough	21824d49da	Merge branch 'bg-add-queue-metrics' of github.com:overleaf/document-updater into bg-add-queue-metrics	2020-04-01 16:04:52 +01:00
Brian Gough	3a8c362fba	add doclines set/del metric	2020-04-01 15:59:25 +01:00
Brian Gough	00b11bda96	use separate loop for pendingUpdates metric	2020-04-01 14:50:55 +01:00
Henry Oswald	c2b050e286	bump redis to 1.0.12	2020-03-31 10:21:50 +01:00
Jakob Ackermann	17c2add0cf	[misc] track redis pub/sub payload sizes on publish	2020-03-30 11:31:43 +02:00
Shane Kilkelly	861ab5be80	update metrics and logger for latest bug fixes	2020-03-26 13:40:47 +00:00
Brian Gough	fcb72b9bf7	update tests	2020-03-25 14:27:41 +00:00
Brian Gough	1a0550364d	add metric for getdoc bytes	2020-03-25 14:27:41 +00:00
Brian Gough	891fcc696b	add metric for pending updates queue	2020-03-25 14:27:41 +00:00
Brian Gough	e293d86c14	add metric for project history queue	2020-03-25 12:15:16 +00:00
Jakob Ackermann	b2d1718a2e	[misc] bump logger-sharelatex to 1.9.1	2020-03-23 16:18:05 +01:00
Shane Kilkelly	ada4fba3dc	Fix express deprecations	2020-03-19 15:39:57 +00:00
Shane Kilkelly	21194e52ca	more upgrades	2020-03-19 15:23:45 +00:00
Shane Kilkelly	f069a2515d	upgrade express	2020-03-19 15:20:32 +00:00
Shane Kilkelly	aa324a1920	npm audit fix	2020-03-19 15:08:59 +00:00
Shane Kilkelly	b631a741ac	update build scripts	2020-03-19 15:06:36 +00:00
Brian Gough	e2564c9cdd	update metrics module to 2.5.1 ... reduce time window for summary metrics to 60s	2020-03-11 10:25:48 +00:00
Brian Gough	69125df44b	Merge pull request #121 from overleaf/bg-clean-up-package-json ... clean up package.json	2020-03-11 10:20:30 +00:00
Eric Mc Sween	ff32104fe6	Merge pull request #123 from overleaf/em-doc-hard-delete ... Add ignore_flush_errors option to the doc delete endpoint	2020-03-10 10:11:00 -04:00
Eric Mc Sween	d9caced0d6	Change skip_flush option to ignore_flush_errors in delete doc ... Instead of skipping the flush, we'll still try to flush and proceed with the doc deletion, even when the flush fails.	2020-03-10 09:40:49 -04:00
Eric Mc Sween	9b70eb75b3	Rename flush param to skip_flush in delete doc ... Also move it to the query string instead of the body.	2020-03-09 16:27:32 -04:00
Eric Mc Sween	c09bc0e868	Add a "flush: false" option to the doc delete endpoint ... This will delete the document from Redis without flushing to web, docstore or history. To be used when something is broken.	2020-03-07 08:59:15 -05:00
nate stemen	ffd8d0745d	use empty object for ranges if it doesn't exist	2020-03-06 13:49:30 -05:00
Brian Gough	852ae5fbae	move bunyan out of devDependencies ... (needed by @google-cloud/logging-bunyan)	2020-02-26 09:47:38 +00:00
Brian Gough	0419039d4d	Merge branch 'master' into bg-use-separate-redis-for-project-history	2020-02-21 14:13:33 +00:00
Brian Gough	2e178b0e2d	resolve merge conflicts	2020-02-21 14:06:58 +00:00
Brian Gough	922f237c39	Revert "add redislabs ca cert to repository" ... This reverts commit eb7419b0f45992228393086caf7ed6f66296801f.	2020-02-19 09:26:59 +00:00
Brian Gough	412eabc306	Revert "add tls settings" ... This reverts commit 72a4994cebab2731f99f0ada7a094c8a0acb3293.	2020-02-19 09:26:42 +00:00
Brian Gough	4e7c5c9046	update requestretry	2020-02-18 09:29:53 +00:00
Brian Gough	3aa997a0bc	update package-lock.json	2020-02-17 14:06:13 +00:00
Brian Gough	d9138c91f4	upgrade request to latest version	2020-02-17 14:05:59 +00:00
Brian Gough	583af0bfc4	update package-lock.json	2020-02-17 13:42:57 +00:00
Brian Gough	638688a854	remove unused lynx module	2020-02-17 13:41:53 +00:00
Brian Gough	908e916779	move sinon and sandboxed-module to devDependencies	2020-02-17 13:41:34 +00:00
Brian Gough	dec18e0773	Merge pull request #120 from overleaf/dependabot/npm_and_yarn/lodash-4.17.13 ... Bump lodash from 4.17.4 to 4.17.13	2020-02-17 11:09:49 +00:00
Brian Gough	0969a39291	Merge pull request #119 from overleaf/dependabot/npm_and_yarn/sshpk-1.16.1 ... Bump sshpk from 1.13.1 to 1.16.1	2020-02-17 11:02:45 +00:00
Brian Gough	30567ca2da	Merge pull request #118 from overleaf/dependabot/npm_and_yarn/stringstream-0.0.6 ... Bump stringstream from 0.0.5 to 0.0.6	2020-02-17 10:59:59 +00:00
dependabot[bot]	9a92cd7b31	Bump lodash from 4.17.4 to 4.17.13 ... Bumps [lodash](https://github.com/lodash/lodash) from 4.17.4 to 4.17.13. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](https://github.com/lodash/lodash/compare/4.17.4...4.17.13) Signed-off-by: dependabot[bot] <support@github.com>	2020-02-14 09:56:26 +00:00
dependabot[bot]	3fb0ed414c	Bump sshpk from 1.13.1 to 1.16.1 ... Bumps [sshpk](https://github.com/joyent/node-sshpk) from 1.13.1 to 1.16.1. - [Release notes](https://github.com/joyent/node-sshpk/releases) - [Commits](https://github.com/joyent/node-sshpk/compare/v1.13.1...v1.16.1) Signed-off-by: dependabot[bot] <support@github.com>	2020-02-14 09:56:25 +00:00
dependabot[bot]	2acd4c0f23	Bump stringstream from 0.0.5 to 0.0.6 ... Bumps [stringstream](https://github.com/mhart/StringStream) from 0.0.5 to 0.0.6. - [Release notes](https://github.com/mhart/StringStream/releases) - [Commits](https://github.com/mhart/StringStream/compare/v0.0.5...v0.0.6) Signed-off-by: dependabot[bot] <support@github.com>	2020-02-14 09:56:24 +00:00
dependabot[bot]	c4bec0dcc7	Bump extend from 3.0.1 to 3.0.2 ... Bumps [extend](https://github.com/justmoon/node-extend) from 3.0.1 to 3.0.2. - [Release notes](https://github.com/justmoon/node-extend/releases) - [Changelog](https://github.com/justmoon/node-extend/blob/master/CHANGELOG.md) - [Commits](https://github.com/justmoon/node-extend/compare/v3.0.1...v3.0.2) Signed-off-by: dependabot[bot] <support@github.com>	2020-02-14 09:56:23 +00:00
Jakob Ackermann	5fb8c0a098	Merge pull request #116 from overleaf/jpa-dependencies-cleanup ... [misc] rename npm-shrinkwrap.json to package-lock.json and run npm i	2020-02-14 10:55:54 +01:00
Jakob Ackermann	af4211d167	[misc] rename npm-shrinkwrap.json to package-lock.json and run npm i	2020-02-12 14:39:51 +01:00
Brian Gough	81e21c5e76	remove unused .travis.yml file	2020-02-12 12:37:00 +00:00
Jakob Ackermann	82ab2f7d14	Merge pull request #65 from das7pad/hotfix-redis-config ... [redis] use the correct instance for the real time updates	2020-02-12 13:21:04 +01:00
Jakob Ackermann	e73978410c	Merge pull request #67 from das7pad/hotfix-static-queue-check ... [misc] tests: fix a static test	2020-02-12 13:20:55 +01:00
Jakob Ackermann	776d85b5b0	Merge pull request #114 from overleaf/jpa-build-scripts-1-3-5 ... [misc] update the build scripts to 1.3.5	2020-02-12 13:20:44 +01:00
Brian Gough	26d8d07e7a	remove unused redis settings from rate limit unit test	2020-02-11 14:32:49 +00:00
Jakob Ackermann	9cfc59734a	[misc] update the build scripts to 1.3.5	2020-02-11 12:27:53 +01:00
Brian Gough	2d2e3bc2b1	Merge pull request #111 from overleaf/bg-add-comment-about-null-bytes ... add comment about null byte check	2020-02-10 10:26:24 +00:00
Brian Gough	afe43fa252	update to node 10.19.0	2020-02-07 14:16:31 +00:00
Jakob Ackermann	4070b0ef6f	[misc] tests: fix a static test ... `ProjectHistoryRedisManager.queueOps` is masked and can not push changes into redis during the unittest. Signed-off-by: Jakob Ackermann <das7pad@outlook.com>	2020-02-07 12:36:46 +01:00
Jakob Ackermann	10d177f41b	[misc] test/unit: stub the correct redis settings	2020-02-07 12:24:57 +01:00
Brian Gough	338d3609f5	add comment about null byte check	2020-01-30 15:17:13 +00:00
Brian Gough	544ae05212	added note about rollback	2020-01-23 16:22:26 +00:00
Brian Gough	626e19ed1a	add logging of migration phase at startup	2020-01-23 15:46:54 +00:00
Brian Gough	d5a2b96df9	add note about deleting the migration key entries	2020-01-23 14:36:59 +00:00
Jakob Ackermann	fcfa3ecc9d	[misc] upgrade node to 10.18.1	2020-01-23 10:05:57 +01:00
Brian Gough	31324fb65a	add environment variable for migration_phase setting	2020-01-22 15:36:06 +00:00
Brian Gough	ad58fe76b2	add tls settings	2020-01-21 15:36:33 +00:00
Brian Gough	531d9b77b9	add redislabs ca cert to repository	2020-01-21 09:49:41 +00:00
Brian Gough	7036803acf	add missing argument to metrics.inc ... also track retries rather than attempts (which is always 1 for a successful request)	2020-01-14 15:00:21 +00:00
Brian Gough	3caa0e7c05	add failure/retry metrics for web-api requests	2020-01-14 13:53:50 +00:00
Brian Gough	a638ef4251	add comment about locking in redis migration	2020-01-13 15:56:28 +00:00
Brian Gough	c2714f9ae9	add acceptance tests for RedisMigrationManager	2020-01-06 16:50:06 +00:00
Brian Gough	27044c2d02	allow migration phase to be modified at runtime for testing	2020-01-06 16:46:35 +00:00
Brian Gough	8ae95ebf60	fix rclient check in migration metrics	2020-01-06 16:45:36 +00:00
Brian Gough	97cbf46160	add metrics for migration	2019-12-16 11:46:35 +00:00
Brian Gough	a2e63d009e	fix migration phase check	2019-12-16 09:55:26 +00:00
Brian Gough	a85dffbcef	fix acceptance tests	2019-12-16 09:27:00 +00:00
Brian Gough	d0c5eb5698	support migration of project history keys to separate redis instance	2019-12-13 16:38:41 +00:00
Brian Gough	1bb12a5035	allow pending updates to clear in acceptance tests	2019-12-11 14:43:59 +00:00
Brian Gough	ad19fee667	add setting so that double flush is the default ... can be disabled to stop flushing to track-changes	2019-11-25 13:36:25 +00:00
Brian Gough	4f6583bbf2	fix getDocVersion and add tests	2019-11-25 13:28:36 +00:00
Brian Gough	68e12f4d2d	add metrics for queue operations	2019-11-25 10:51:10 +00:00
Brian Gough	8b73bb9f13	Merge branch 'master' into bg-filter-track-changes-updates	2019-11-22 10:41:33 +00:00
Brian Gough	b7055eecee	add metrics for history flushes	2019-11-22 09:14:32 +00:00
Brian Gough	65cf4cf7c7	make flush to track-changes failsafe	2019-11-21 14:58:35 +00:00
Brian Gough	dcd7649bad	filter track-changes updates for projects using project-history	2019-11-19 10:02:56 +00:00
Brian Gough	08ee58abb8	remove console.log from tests	2019-11-15 09:34:13 +00:00
Brian Gough	82a72c9b73	fix missing bodyParser limit	2019-11-14 16:32:59 +00:00
dependabot[bot]	6871889bfb	Bump request from 2.25.0 to 2.47.0 ... Bumps [request](https://github.com/request/request) from 2.25.0 to 2.47.0. - [Release notes](https://github.com/request/request/releases) - [Changelog](https://github.com/request/request/blob/master/CHANGELOG.md) - [Commits](https://github.com/request/request/compare/v2.25.0...v2.47.0) Signed-off-by: dependabot[bot] <support@github.com>	2019-11-05 11:38:25 +00:00
dependabot[bot]	ccc072e9da	Bump express from 3.3.4 to 3.11.0 ... Bumps [express](https://github.com/expressjs/express) from 3.3.4 to 3.11.0. - [Release notes](https://github.com/expressjs/express/releases) - [Changelog](https://github.com/expressjs/express/blob/master/History.md) - [Commits](https://github.com/expressjs/express/compare/3.3.4...3.11.0) Signed-off-by: dependabot[bot] <support@github.com>	2019-10-30 13:38:50 +00:00
Nate Stemen	1d3a1d22be	bump build script to 1.1.24	2019-10-25 12:50:45 -04:00
Nate Stemen	1620956e2e	replace private link with public one	2019-10-25 12:50:33 -04:00
Brian Gough	a9428c9ece	Merge pull request #96 from overleaf/bg-add-ioredis-maxretriesperrequest ... add ioredis maxRetriesPerRequest	2019-10-16 08:55:36 +01:00
Brian Gough	21a9126847	set maxRetriesPerRequest from environment variable	2019-10-16 08:44:08 +01:00
Brian Gough	781e01b8b4	Merge pull request #95 from overleaf/bg-update-ioredis ... update to ioredis 4.14.1	2019-10-14 09:21:39 +01:00
Brian Gough	5a62632cfd	add ioredis option maxRetriesPerRequest: 0 ... see https://github.com/luin/ioredis/issues/965	2019-10-04 11:53:14 +01:00
Brian Gough	51a821c03c	remove old unused ioredis keepalive option	2019-10-04 11:51:37 +01:00
Brian Gough	18ccd11224	update to ioredis 4.14.1	2019-10-04 11:32:08 +01:00
Henry Oswald	e9efd6b93b	logs out when a redis event occurs	2019-10-03 11:10:31 +01:00
Brian Gough	d82b180b76	avoid project history queues building up with deferred flush	2019-10-03 04:05:24 +01:00
Brian Gough	c1454bc4ac	Merge pull request #92 from overleaf/bg-flush-queue-prod-fixes ... add continuous background flush	2019-10-02 13:11:00 +01:00
Brian Gough	0c14b7d2f8	add comment about background flush limit	2019-10-01 15:06:01 +01:00
Brian Gough	ae3ebf2db6	start background flush after http server has started	2019-10-01 15:02:37 +01:00
Brian Gough	2845b23b70	add smoothing of delete spikes	2019-10-01 15:01:53 +01:00
Brian Gough	2c22a60052	add random jitter to cutoff time	2019-10-01 15:01:20 +01:00
Brian Gough	a32495d2b4	make background flush more adaptive	2019-10-01 14:09:41 +01:00
Brian Gough	73b4262186	add continuous background flush	2019-09-30 16:05:53 +01:00
Brian Gough	33fadf51c1	fix getDocTimestamps for multiple docs	2019-09-30 13:50:25 +01:00
Brian Gough	89a90399fe	Merge pull request #89 from overleaf/bg-queue-deletes ... queue deletes for deferred processing	2019-09-30 11:14:08 +01:00
John Lees-Miller	c5a9105c33	Update config	2019-09-28 11:07:33 +01:00
Brian Gough	260923f291	keep flushQueuedProjects in the foreground	2019-09-27 10:46:24 +01:00
Brian Gough	7561e05660	check timestamps array length	2019-09-27 10:39:56 +01:00
Brian Gough	0f0682df43	allow flush to complete in acceptance test	2019-09-26 16:00:11 +01:00
Brian Gough	b7f3b848af	remove unused dryRun option ... Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>	2019-09-26 15:50:55 +01:00
Brian Gough	3bc176259b	fix log line	2019-09-26 15:46:54 +01:00
Brian Gough	8cdc8c410a	fix error logging	2019-09-26 15:46:45 +01:00
Brian Gough	fc62abfcfa	run flush of queued projects in the background	2019-09-26 15:46:14 +01:00
Brian Gough	ba35c73cb6	add comment about ZPOPMIN	2019-09-26 15:18:10 +01:00
Brian Gough	a709a0adaa	for simplicity keep the cutoff time the same while flushing the queue	2019-09-26 15:05:38 +01:00
Brian Gough	eae4b352ca	remove unnecessary check	2019-09-26 14:59:03 +01:00
Brian Gough	9f358ead9f	add an acceptance test for flush with queue processing	2019-09-26 10:55:05 +01:00
Brian Gough	b49621b3e9	add comments	2019-09-26 10:14:49 +01:00
Brian Gough	83dd43b809	add metric for queue length	2019-09-25 17:04:36 +01:00
Brian Gough	f6b2ac7360	queue deletes for deferred processing	2019-09-25 16:42:49 +01:00
Brian Gough	c32101f0fb	upgrade redis-sharelatex to 1.0.10	2019-09-19 16:22:43 +01:00
Brian Gough	912a3a7753	remove redis server-side hashing for performance ... we still compute the document hash in node, and check it on retrieval but we don't check the hash at the point of writing it in redis which was previously done with a redis Lua script.	2019-09-09 15:27:58 +01:00
Henry Oswald	aa15a76059	added log lines for all project flushing	2019-08-30 07:38:53 +01:00
Henry Oswald	0ae838dd2d	add logger into project flusher	2019-08-29 20:36:00 +01:00
Brian Gough	e75f2cc325	add unit tests for skipping history flush	2019-08-15 11:01:45 +01:00
Brian Gough	7493462154	add acceptance test for realtime shutdown	2019-08-15 10:38:07 +01:00
Brian Gough	a76e0dca88	skip history flush when project is cleared by realtime shutdown ... history is flushed by a background cron job anyway	2019-08-15 09:51:16 +01:00
Simon Detheridge	f49f5cbf05	Merge pull request #79 from overleaf/ho-mongo-connection-string ... use MONGO_CONNECTION_STRING in settings.defaults.coffee if set	2019-08-08 14:16:49 +01:00
Simon Detheridge	6721b904a7	Merge pull request #82 from overleaf/bg-mongo-health-check ... add a combined health check for mongo and redis	2019-08-08 14:16:38 +01:00
Simon Detheridge	06444d2cc4	Improve/fix serializers for update logging (#80) ... * Improve/fix serializers for update logging	2019-08-08 14:10:54 +01:00
Brian Gough	40f6494b19	add a combined health check for mongo and redis	2019-08-07 16:25:23 +01:00
Henry Oswald	aa1c74a979	use MONGO_CONNECTION_STRING in settings.defaults.coffee if set	2019-08-07 15:44:57 +01:00
Simon Detheridge	df9ca8b272	Add serializer to print only length of large fields in production	2019-07-31 16:42:28 +01:00
Brian Gough	618880f99d	remove unnecessary check for doc_id	2019-07-24 16:57:43 +01:00
Brian Gough	c9ccf62d71	support per-doc pubsub channels	2019-07-22 12:20:06 +01:00
Brian Gough	97487a077e	fix cluster/sentinel connection with real-time	2019-07-10 09:42:05 +01:00
Henry Oswald	06ad0f7acd	Merge pull request #75 from overleaf/ho-pubsub-connection ... Remove real time redis connection and consolidate on pubsub	2019-07-08 13:58:41 +01:00
Henry Oswald	374585f47d	Merge pull request #72 from overleaf/sk-dep-upgrades-2019-06 ... Update logger, metrics, and redis	2019-07-08 13:58:28 +01:00
Henry Oswald	3b3b2da0f5	add pubsub redis connection and remove real time redis connection	2019-07-04 13:34:31 +01:00
Henry Oswald	eed8e215d7	bump redis-sharelatex (and io redis) to 1.0.9	2019-07-02 16:58:30 +01:00
Shane Kilkelly	e8a8c446ed	Increase timeout on test?	2019-06-19 11:34:38 +01:00
Shane Kilkelly	080b482e51	Update logger, metrics, and redis ... Also fix acceptance tests, broken by a change in the redis driver behaviour. It now returns promises from most operations, which confuses mocha.	2019-06-19 10:25:54 +01:00
Henry Oswald	fdef197271	Merge branch 'master' into ho-detailed-flush-status	2019-06-13 14:33:22 +01:00
Henry Oswald	d9a737f97c	return failed and succesfully flushed projects when flushing everything	2019-06-13 14:21:38 +01:00
Brian Gough	e8dd1aae9c	Merge pull request #70 from overleaf/bg-metric-for-invalid-hash ... add metric for invalid hash and other sharejs errors	2019-06-12 13:50:34 +01:00
Brian Gough	d50b93df2f	add metric for invalid hash and other sharejs errors	2019-06-11 16:48:06 +01:00
Brian Gough	984b2d38e6	add unit tests	2019-06-11 16:33:14 +01:00
Brian Gough	966478cac4	fix hash check to use 'v' field instead of version	2019-06-11 14:11:46 +01:00
Brian Gough	e95059f98e	handle non-urgent flushes in background	2019-06-03 10:01:10 +01:00
Brian Gough	0bbfa7de27	Merge branch 'master' into bg-downgrade-delete-component-error	2019-05-08 09:07:02 +01:00
Henry Oswald	d169cca425	Merge branch 'master' into ho-flush-all-projects	2019-05-07 17:02:14 +01:00
Brian Gough	27a8248196	convert "Delete component" errors into warnings	2019-05-07 16:55:17 +01:00
Tim Alby	8b40da701e	update README ... - remove build status badge - change app name - update copyright notice - update links	2019-05-07 17:03:09 +02:00
Tim Alby	d316f172bf	update repo URL for Jenkins	2019-05-07 17:03:06 +02:00
Henry Oswald	e57741cb80	stub out project flusher for unit tests	2019-05-07 15:46:30 +01:00
Henry Oswald	4b8a27a220	change github url	2019-05-07 14:01:12 +01:00
Tim Alby	ca6bfd8929	Update README	2019-05-06 17:28:52 +02:00
Henry Oswald	daca83a057	add dryRun option to flush all projects	2019-05-02 16:54:22 +01:00
Henry Oswald	d5d1736a5e	adds /flush_all_projects project	2019-05-02 16:43:35 +01:00
Tim Alby	c1c23e4bee	record last author id on document flush ... This is a multi-steps process: * get a update's `user_id` from the metadata * store the `user_id` (`lastUpdatedBy`) and current date (`lastUpdatedAt`) for the document in Redis on every updates * fetch `lastUpdatedAt` and `lastUpdatedBy` from Redis on document flush * send the data to web to be persisted in Mongo	2019-05-02 11:10:02 +01:00
Brian Gough	68e7b9c4e9	Merge pull request #48 from sharelatex/bg-check-incoming-hash ... check incoming hash when present	2019-04-29 10:15:44 +01:00
James Allen	52f3596e53	Review feedback	2019-04-16 11:05:17 +01:00
James Allen	33478f95fd	Fix package.json versions	2019-04-11 16:32:31 +01:00
James Allen	3d76f4b9bf	Record a snapshot to mongo when a doc's comments/changes get collapsed	2019-04-11 13:27:46 +01:00
Brian Gough	3c635c8d98	check version before it is modified by applyOp	2019-04-09 09:20:48 +01:00
Brian Gough	cc1f3fce5b	check incoming hash when present	2019-04-08 14:12:18 +01:00
Brian Gough	fd1425d83f	include a unique id in every message published to redis	2019-03-21 12:10:15 +00:00
Henry Oswald	4d7f9f3c1a	add sentry into settings.defaults	2019-03-05 17:34:27 +00:00
Brian Gough	11f07d1f09	Merge pull request #45 from sharelatex/bg-avoid-text-html-content-type-in-responses ... use explicit json content-type to avoid security issues with text/html	2019-02-22 09:24:30 +00:00
Henry Oswald	937e8fe071	use redis cluster key in defaults for pendingupdates	2019-02-15 13:58:29 +00:00
Brian Gough	8c5d74faef	use explicit json content-type to avoid security issues with text/html	2019-02-12 16:45:11 +00:00
Henry Oswald	0a0fc91f28	Update app.coffee	2019-02-12 13:27:13 +00:00
Henry Oswald	3bc4cb492a	added log line	2019-02-07 16:30:53 +00:00
Henry Oswald	08723f8972	revert health check redis types	2019-02-07 15:53:26 +00:00
Henry Oswald	ecaef6485b	revert the removal of realtime keyspace	2019-02-07 15:27:51 +00:00
Henry Oswald	b5564095f3	add /health_check/redis route back in	2019-02-07 15:10:40 +00:00
Henry Oswald	73bd264401	remove realtime keys in settings, no longer used	2019-02-07 14:55:24 +00:00
Henry Oswald	2998750a33	fix redis version lock	2019-02-06 16:01:44 +00:00
Henry Oswald	4e1a2c787c	Revert "turn down logging, use logger.info for less important data" ... This reverts commit c5f91428e3c7702fbbd3ffd1ef7a772d513f33f2.	2019-02-06 15:29:22 +00:00
Henry Oswald	8d3fb729c5	point to web_api	2019-02-01 20:10:47 +00:00
Henry Oswald	e44a02391a	standardise name of document-updater	2019-02-01 20:04:43 +00:00
Henry Oswald	98de529e42	make config look like chef config	2019-02-01 19:52:00 +00:00
Henry Oswald	3051d7db4b	bulk upgrade to 1.1.12, moved to npm rather than git	2019-01-31 16:00:24 +00:00
Henry Oswald	d279a9149c	add dispatcherCount to config file	2019-01-25 17:24:19 +00:00
Henry Oswald	76b054c7f4	bump node redis to 1.0.5	2019-01-25 17:01:59 +00:00
Christopher Hoskin	3eb2d27d4a	Add **/*.map to .gitignore	2019-01-08 15:01:48 +00:00
Christopher Hoskin	5db1913cec	Bump settings to v1.1.0	2019-01-08 15:00:26 +00:00
Christopher Hoskin	a783c6b3cc	Bump logger to 1.5.9	2019-01-04 09:33:08 +00:00
Christopher Hoskin	05a80c7cba	Add app.map to .gitignore	2019-01-04 09:31:00 +00:00
Christopher Hoskin	0aa7315c27	Bump buildscripts to 1.1.10	2019-01-04 09:24:09 +00:00
Christopher Hoskin	d9ec90f0e2	Move to metrics v2 and shrinkwrap	2019-01-04 09:22:09 +00:00
Christopher Hoskin	1217d8a80a	Merge branch 'master' into csh-ho-docker-issue-1338-bulk-upgrade	2019-01-04 09:18:40 +00:00
Henry Oswald	2505be283a	turn down logging, use logger.info for less important data	2018-12-13 12:46:10 +00:00
Henry Oswald	25e1ad7491	remove redis.realtime ... real time pushes data into the document updater redis instance	2018-12-05 16:17:41 +00:00
Henry Oswald	9437cf8b02	explicity split redis config values	2018-12-05 15:16:55 +00:00
Henry Oswald	5f046ed329	install metrics route and bump metrics version	2018-12-05 12:35:46 +00:00
Brian Gough	aae1352519	ensure that project history is flushed when the project is deleted	2018-10-30 12:03:44 +00:00
Henry Oswald	a11d6dcc48	Merge branch 'master' into ho-docker	2018-10-15 15:17:40 +01:00
Alasdair Smith	c128e0ab1c	Use setting instead of hard-coding port	2018-10-09 11:46:27 +01:00
Henry Oswald	ff673c71d5	set redis port via env var	2018-09-28 17:04:34 +01:00
Henry Oswald	93cf574251	upgrade build scripts 1.1.9	2018-09-28 15:13:47 +01:00
Henry Oswald	3db7377d74	add process.env["REDIS_PASSWORD"]	2018-09-28 15:11:17 +01:00
Brian Gough	537bd9a0c0	Merge pull request #36 from sharelatex/bg-limit-resync-requests-to-web ... limit resync requests to web (connects to overleaf/sharelatex#775)	2018-08-16 12:25:16 +01:00
Brian Gough	910b27357d	add more logging to resync operations	2018-08-16 11:14:11 +01:00
Brian Gough	aa013f0bee	limit parallel resync doc requests to web	2018-08-16 11:13:11 +01:00
Brian Gough	fdcb806518	set a timestamp for the first entry in the projectHistory:Ops queue	2018-07-24 09:17:31 +01:00
Brian Gough	e471730efb	Revert "Merge pull request #33 from sharelatex/bg-add-timestamp-marker-to-project-history-queue" ... This reverts commit 8ec398b133aa75c45384dd7ceb630cd743f4a15c, reversing changes made to e3a6c934cf53fd245d7d0df737463cc90f954885.	2018-07-23 16:05:04 +01:00
Brian Gough	33e898f671	update to 1.1.9 build scripts	2018-07-23 15:35:54 +01:00
Brian Gough	6237577f87	set a timestamp for the first entry in the projectHistory:Ops queue	2018-07-20 10:43:31 +01:00
Hayden Faulds	62a10f25f2	move tk call to before sandboxed module call	2018-06-05 13:54:30 +01:00
Henry Oswald	581963a4af	Dockerised env. 1.1.3 build scripts	2018-05-23 13:52:20 +01:00
Brian Gough	7e526395fb	fix check for "delete component" message	2018-04-30 09:56:01 +01:00
Brian Gough	37f431f148	also block "delete component" messages from sentry	2018-04-27 16:03:46 +01:00
Brian Gough	64670f272a	stop OpRangeNotAvailable errors going to sentry	2018-04-27 15:45:28 +01:00
Hayden Faulds	af92ca70a1	coerce projectHistoryId to integer after reading from Redis	2018-04-23 15:19:06 +01:00
Hayden Faulds	1b63141e49	Revert "Revert "Add projectHistoryId to updates""	2018-04-23 12:08:04 +01:00
Hayden Faulds	3cc20d6311	Revert "Add projectHistoryId to updates"	2018-04-23 11:41:29 +01:00
Hayden Faulds	9629d0a11e	Merge pull request #26 from sharelatex/hof-deleted-project-history ... Add projectHistoryId to updates	2018-04-23 11:01:53 +01:00
Alberto Fernández Capel	7d597fe59e	Do not specify node version in .travis.yml ... So travis picks instead the version from .nvmrc	2018-04-19 14:03:35 +01:00
Hayden Faulds	25c0be4eaa	add projectHistryId to project structure updates	2018-04-13 15:22:05 +01:00
Hayden Faulds	a220794d32	add projectHistoryId to resync updates	2018-04-13 15:22:05 +01:00
Hayden Faulds	fb1852a593	cache projectHistoryId with doc in Redis	2018-04-13 15:22:05 +01:00
Brian Gough	dd0f8b880a	change update to projectUpdate in project related methods	2018-03-16 10:54:12 +00:00
Brian Gough	779f00f912	add acceptance test for ordering of project structure changes	2018-03-15 16:27:14 +00:00
Brian Gough	3385d2640a	fix structure ordering bug	2018-03-15 16:27:10 +00:00
Brian Gough	75a5428cbf	update acceptance tests	2018-03-15 16:24:14 +00:00
Brian Gough	b3887fd984	update unit tests for incoming project versions	2018-03-15 16:24:11 +00:00
Brian Gough	0642e3c8c9	support project version on incoming requests	2018-03-15 16:15:10 +00:00
Hayden Faulds	4a5731440a	Merge pull request #21 from sharelatex/hof-history-sync ... resync project history endpoint	2018-03-14 10:40:59 +00:00
Hayden Faulds	1226f96fc3	extract ProjectHistoryRedisManager	2018-03-09 15:36:38 +00:00
Hayden Faulds	fc6ef6ea7a	improve logging	2018-03-09 12:21:48 +00:00
Hayden Faulds	088ab6e7ba	rename queueResyncDocContents -> queueResyncDocContent	2018-03-09 12:21:37 +00:00
Hayden Faulds	64d1fe13bc	additional logging for project history resync	2018-03-07 16:24:22 +00:00
Hayden Faulds	a72d228354	add unit tests	2018-03-07 13:29:53 +00:00
Hayden Faulds	152a25e96d	rename resyncProject -> resyncProjectHistory	2018-03-07 13:29:42 +00:00
Hayden Faulds	039f997e80	update project history resync url	2018-03-07 11:37:26 +00:00
Hayden Faulds	d46217be52	queue doc content sync updates	2018-03-02 15:02:27 +00:00
Hayden Faulds	52d6b710bd	add ability to resync project structure	2018-03-02 15:01:53 +00:00
James Allen	939eaa2d4b	Don't allow a document to be loaded without a pathname	2018-03-02 10:02:49 +00:00
Hayden Faulds	5a11332aa3	remove unecessary method wrapping	2018-02-22 10:17:59 +00:00
Hayden Faulds	0f87ae1f74	simplify comments	2018-02-22 10:16:29 +00:00
Hayden Faulds	ea0dd9700b	reduce the number of times we flush project history	2018-02-22 10:01:05 +00:00
James Allen	3d5740fd7d	Merge pull request #18 from sharelatex/msw-flush-project-ops ... Flush project structure changes every 500	2018-02-16 10:30:00 +00:00
James Allen	5425fdef8a	update to 1.0.1 build scripts	2018-02-15 16:54:21 +00:00
James Allen	657da70d45	Fix unit test namespacing from mocha upgrade	2018-02-15 16:28:58 +00:00
James Allen	5d1659457e	Update to build 1.0.0 build scripts	2018-02-15 16:28:40 +00:00
James Allen	f6c79f3203	Fix and extend unit tests	2018-02-15 15:55:12 +00:00
Michael Walker	6c57317f8d	Add a test that sending too few updates does not flush history	2018-01-31 11:41:08 +00:00
Michael Walker	241d1b27d5	Remove _shouldFlushHistoryOps wrapper	2018-01-31 11:27:40 +00:00
Michael Walker	772ee4083a	Turn a unit test into an acceptance test	2018-01-31 11:17:56 +00:00
Michael Walker	bdaa2ffa77	Add an acceptance test for flushing project changes	2018-01-26 14:41:47 +00:00
Michael Walker	d2865f0650	Correctly count project_ops_length	2018-01-26 14:41:28 +00:00
Michael Walker	3028fb9c3d	Only flush project updates when crossing the threshold	2018-01-24 11:57:03 +00:00
Michael Walker	24c74db0dc	Flush project changes when queue hits limit	2018-01-23 11:18:20 +00:00
Michael Walker	3d313ebc53	Add method to get number of queued project updates	2018-01-23 11:10:54 +00:00
James Allen	ccf6cb1a18	Provide hosts as environment settings, add npm run start script and update coffeescript	2017-12-29 08:13:16 +00:00
Brian Gough	0b583dd4e6	fix log message	2017-12-20 13:27:10 +00:00
Brian Gough	a8b14552c0	allow flush to succeed when doc is already deleted	2017-12-19 12:27:57 +00:00
Brian Gough	41f15c4fa8	fix unit test	2017-12-14 14:46:27 +00:00
Brian Gough	947e18eccf	Merge branch 'bg-limit-ops-in-lock'	2017-12-14 14:26:44 +00:00
James Allen	6abf9c227d	Calculate and send doc length before each update	2017-12-11 17:26:32 +00:00
Hayden Faulds	bd45fcb5f8	Merge pull request #12 from sharelatex/hof-version-entity-creation ... version entity creation; Connects to sharelatex/web-sharelatex-internal#104	2017-11-24 10:02:51 +00:00
Brian Gough	fb572670cd	Merge pull request #11 from sharelatex/bg-use-separate-flush-count-for-projects ... use a separate parameter for doc and project flush	2017-11-21 13:54:03 +00:00
Brian Gough	0bb9f147aa	fix whitespace	2017-11-13 11:56:08 +00:00
Brian Gough	a4bbf5cf52	use a separate parameter for doc and project flush	2017-11-13 11:53:39 +00:00
Hayden Faulds	d0e56e5054	add acceptance tests for entity additions	2017-11-10 15:01:37 +00:00
Hayden Faulds	a3420b1236	version entity additions	2017-11-10 14:54:56 +00:00
Hayden Faulds	944e633bac	add acceptance test for entity renaming	2017-11-08 09:20:38 +00:00
Hayden Faulds	7e86afe55e	version file renames	2017-11-08 09:20:38 +00:00
Hayden Faulds	6d571e6d23	version document renames	2017-11-08 09:20:37 +00:00
Hayden Faulds	b8052e7612	allow settings.redis.project_history to be undefined	2017-10-25 13:56:38 +01:00
Hayden Faulds	5b7be93c79	Merge pull request #2 from sharelatex/hof-doc-pathname ... Add metadata to updates for history	2017-10-25 13:13:00 +01:00