github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-21 20:47:08 -05:00
Code Issues Projects Releases Packages Wiki Activity
834 commits 3 branches 7 tags 206 MiB
Commit graph

723 commits

Author SHA1 Message Date
Brian Gough
30567ca2da Merge pull request #118 from overleaf/dependabot/npm_and_yarn/stringstream-0.0.6 
Bump stringstream from 0.0.5 to 0.0.6
 2020-02-17 10:59:59 +00:00
dependabot[bot]
9a92cd7b31 Bump lodash from 4.17.4 to 4.17.13 
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.4 to 4.17.13.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.4...4.17.13)

Signed-off-by: dependabot[bot] <support@github.com>
 2020-02-14 09:56:26 +00:00
dependabot[bot]
3fb0ed414c Bump sshpk from 1.13.1 to 1.16.1 
Bumps [sshpk](https://github.com/joyent/node-sshpk) from 1.13.1 to 1.16.1.
- [Release notes](https://github.com/joyent/node-sshpk/releases)
- [Commits](https://github.com/joyent/node-sshpk/compare/v1.13.1...v1.16.1)

Signed-off-by: dependabot[bot] <support@github.com>
 2020-02-14 09:56:25 +00:00
dependabot[bot]
2acd4c0f23 Bump stringstream from 0.0.5 to 0.0.6 
Bumps [stringstream](https://github.com/mhart/StringStream) from 0.0.5 to 0.0.6.
- [Release notes](https://github.com/mhart/StringStream/releases)
- [Commits](https://github.com/mhart/StringStream/compare/v0.0.5...v0.0.6)

Signed-off-by: dependabot[bot] <support@github.com>
 2020-02-14 09:56:24 +00:00
dependabot[bot]
c4bec0dcc7 Bump extend from 3.0.1 to 3.0.2 
Bumps [extend](https://github.com/justmoon/node-extend) from 3.0.1 to 3.0.2.
- [Release notes](https://github.com/justmoon/node-extend/releases)
- [Changelog](https://github.com/justmoon/node-extend/blob/master/CHANGELOG.md)
- [Commits](https://github.com/justmoon/node-extend/compare/v3.0.1...v3.0.2)

Signed-off-by: dependabot[bot] <support@github.com>
 2020-02-14 09:56:23 +00:00
Jakob Ackermann
5fb8c0a098 Merge pull request #116 from overleaf/jpa-dependencies-cleanup 
[misc] rename npm-shrinkwrap.json to package-lock.json and run npm i
 2020-02-14 10:55:54 +01:00
Jakob Ackermann
af4211d167 [misc] rename npm-shrinkwrap.json to package-lock.json and run npm i 2020-02-12 14:39:51 +01:00
Brian Gough
81e21c5e76 remove unused .travis.yml file 2020-02-12 12:37:00 +00:00
Jakob Ackermann
82ab2f7d14 Merge pull request #65 from das7pad/hotfix-redis-config 
[redis] use the correct instance for the real time updates
 2020-02-12 13:21:04 +01:00
Jakob Ackermann
e73978410c Merge pull request #67 from das7pad/hotfix-static-queue-check 
[misc] tests: fix a static test
 2020-02-12 13:20:55 +01:00
Jakob Ackermann
776d85b5b0 Merge pull request #114 from overleaf/jpa-build-scripts-1-3-5 
[misc] update the build scripts to 1.3.5
 2020-02-12 13:20:44 +01:00
Brian Gough
26d8d07e7a remove unused redis settings from rate limit unit test 2020-02-11 14:32:49 +00:00
Jakob Ackermann
9cfc59734a [misc] update the build scripts to 1.3.5 2020-02-11 12:27:53 +01:00
Brian Gough
2d2e3bc2b1 Merge pull request #111 from overleaf/bg-add-comment-about-null-bytes 
add comment about null byte check
 2020-02-10 10:26:24 +00:00
Brian Gough
afe43fa252 update to node 10.19.0 2020-02-07 14:16:31 +00:00
Jakob Ackermann
4070b0ef6f [misc] tests: fix a static test 
`ProjectHistoryRedisManager.queueOps` is masked and can not push
 changes into redis during the unittest.

Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-02-07 12:36:46 +01:00
Jakob Ackermann
10d177f41b [misc] test/unit: stub the correct redis settings 2020-02-07 12:24:57 +01:00
Brian Gough
338d3609f5 add comment about null byte check 2020-01-30 15:17:13 +00:00
Brian Gough
544ae05212 added note about rollback 2020-01-23 16:22:26 +00:00
Brian Gough
626e19ed1a add logging of migration phase at startup 2020-01-23 15:46:54 +00:00
Brian Gough
d5a2b96df9 add note about deleting the migration key entries 2020-01-23 14:36:59 +00:00
Jakob Ackermann
fcfa3ecc9d [misc] upgrade node to 10.18.1 2020-01-23 10:05:57 +01:00
Brian Gough
31324fb65a add environment variable for migration_phase setting 2020-01-22 15:36:06 +00:00
Brian Gough
ad58fe76b2 add tls settings 2020-01-21 15:36:33 +00:00
Brian Gough
531d9b77b9 add redislabs ca cert to repository 2020-01-21 09:49:41 +00:00
Brian Gough
7036803acf add missing argument to metrics.inc 
also track retries rather than attempts (which is always 1 for a successful request)
 2020-01-14 15:00:21 +00:00
Brian Gough
3caa0e7c05 add failure/retry metrics for web-api requests 2020-01-14 13:53:50 +00:00
Brian Gough
a638ef4251 add comment about locking in redis migration 2020-01-13 15:56:28 +00:00
Brian Gough
c2714f9ae9 add acceptance tests for RedisMigrationManager 2020-01-06 16:50:06 +00:00
Brian Gough
27044c2d02 allow migration phase to be modified at runtime for testing 2020-01-06 16:46:35 +00:00
Brian Gough
8ae95ebf60 fix rclient check in migration metrics 2020-01-06 16:45:36 +00:00
Brian Gough
97cbf46160 add metrics for migration 2019-12-16 11:46:35 +00:00
Brian Gough
a2e63d009e fix migration phase check 2019-12-16 09:55:26 +00:00
Brian Gough
a85dffbcef fix acceptance tests 2019-12-16 09:27:00 +00:00
Brian Gough
d0c5eb5698 support migration of project history keys to separate redis instance 2019-12-13 16:38:41 +00:00
Brian Gough
1bb12a5035 allow pending updates to clear in acceptance tests 2019-12-11 14:43:59 +00:00
Brian Gough
ad19fee667 add setting so that double flush is the default 
can be disabled to stop flushing to track-changes
 2019-11-25 13:36:25 +00:00
Brian Gough
4f6583bbf2 fix getDocVersion and add tests 2019-11-25 13:28:36 +00:00
Brian Gough
68e12f4d2d add metrics for queue operations 2019-11-25 10:51:10 +00:00
Brian Gough
8b73bb9f13 Merge branch 'master' into bg-filter-track-changes-updates 2019-11-22 10:41:33 +00:00
Brian Gough
b7055eecee add metrics for history flushes 2019-11-22 09:14:32 +00:00
Brian Gough
65cf4cf7c7 make flush to track-changes failsafe 2019-11-21 14:58:35 +00:00
Brian Gough
dcd7649bad filter track-changes updates for projects using project-history 2019-11-19 10:02:56 +00:00
Brian Gough
08ee58abb8 remove console.log from tests 2019-11-15 09:34:13 +00:00
Brian Gough
82a72c9b73 fix missing bodyParser limit 2019-11-14 16:32:59 +00:00
dependabot[bot]
6871889bfb Bump request from 2.25.0 to 2.47.0 
Bumps [request](https://github.com/request/request) from 2.25.0 to 2.47.0.
- [Release notes](https://github.com/request/request/releases)
- [Changelog](https://github.com/request/request/blob/master/CHANGELOG.md)
- [Commits](https://github.com/request/request/compare/v2.25.0...v2.47.0)

Signed-off-by: dependabot[bot] <support@github.com>
 2019-11-05 11:38:25 +00:00
dependabot[bot]
ccc072e9da Bump express from 3.3.4 to 3.11.0 
Bumps [express](https://github.com/expressjs/express) from 3.3.4 to 3.11.0.
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/master/History.md)
- [Commits](https://github.com/expressjs/express/compare/3.3.4...3.11.0)

Signed-off-by: dependabot[bot] <support@github.com>
 2019-10-30 13:38:50 +00:00
Nate Stemen
1d3a1d22be bump build script to 1.1.24 2019-10-25 12:50:45 -04:00
Nate Stemen
1620956e2e replace private link with public one 2019-10-25 12:50:33 -04:00
Brian Gough
a9428c9ece Merge pull request #96 from overleaf/bg-add-ioredis-maxretriesperrequest 
add ioredis maxRetriesPerRequest
 2019-10-16 08:55:36 +01:00
Brian Gough
21a9126847 set maxRetriesPerRequest from environment variable 2019-10-16 08:44:08 +01:00
Brian Gough
781e01b8b4 Merge pull request #95 from overleaf/bg-update-ioredis 
update to ioredis 4.14.1
 2019-10-14 09:21:39 +01:00
Brian Gough
5a62632cfd add ioredis option maxRetriesPerRequest: 0 
see https://github.com/luin/ioredis/issues/965
 2019-10-04 11:53:14 +01:00
Brian Gough
51a821c03c remove old unused ioredis keepalive option 2019-10-04 11:51:37 +01:00
Brian Gough
18ccd11224 update to ioredis 4.14.1 2019-10-04 11:32:08 +01:00
Henry Oswald
e9efd6b93b logs out when a redis event occurs 2019-10-03 11:10:31 +01:00
Brian Gough
d82b180b76 avoid project history queues building up with deferred flush 2019-10-03 04:05:24 +01:00
Brian Gough
c1454bc4ac Merge pull request #92 from overleaf/bg-flush-queue-prod-fixes 
add continuous background flush
 2019-10-02 13:11:00 +01:00
Brian Gough
0c14b7d2f8 add comment about background flush limit 2019-10-01 15:06:01 +01:00
Brian Gough
ae3ebf2db6 start background flush after http server has started 2019-10-01 15:02:37 +01:00
Brian Gough
2845b23b70 add smoothing of delete spikes 2019-10-01 15:01:53 +01:00
Brian Gough
2c22a60052 add random jitter to cutoff time 2019-10-01 15:01:20 +01:00
Brian Gough
a32495d2b4 make background flush more adaptive 2019-10-01 14:09:41 +01:00
Brian Gough
73b4262186 add continuous background flush 2019-09-30 16:05:53 +01:00
Brian Gough
33fadf51c1 fix getDocTimestamps for multiple docs 2019-09-30 13:50:25 +01:00
Brian Gough
89a90399fe Merge pull request #89 from overleaf/bg-queue-deletes 
queue deletes for deferred processing
 2019-09-30 11:14:08 +01:00
John Lees-Miller
c5a9105c33 Update config 2019-09-28 11:07:33 +01:00
Brian Gough
260923f291 keep flushQueuedProjects in the foreground 2019-09-27 10:46:24 +01:00
Brian Gough
7561e05660 check timestamps array length 2019-09-27 10:39:56 +01:00
Brian Gough
0f0682df43 allow flush to complete in acceptance test 2019-09-26 16:00:11 +01:00
Brian Gough
b7f3b848af remove unused dryRun option 
Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>
 2019-09-26 15:50:55 +01:00
Brian Gough
3bc176259b fix log line 2019-09-26 15:46:54 +01:00
Brian Gough
8cdc8c410a fix error logging 2019-09-26 15:46:45 +01:00
Brian Gough
fc62abfcfa run flush of queued projects in the background 2019-09-26 15:46:14 +01:00
Brian Gough
ba35c73cb6 add comment about ZPOPMIN 2019-09-26 15:18:10 +01:00
Brian Gough
a709a0adaa for simplicity keep the cutoff time the same while flushing the queue 2019-09-26 15:05:38 +01:00
Brian Gough
eae4b352ca remove unnecessary check 2019-09-26 14:59:03 +01:00
Brian Gough
9f358ead9f add an acceptance test for flush with queue processing 2019-09-26 10:55:05 +01:00
Brian Gough
b49621b3e9 add comments 2019-09-26 10:14:49 +01:00
Brian Gough
83dd43b809 add metric for queue length 2019-09-25 17:04:36 +01:00
Brian Gough
f6b2ac7360 queue deletes for deferred processing 2019-09-25 16:42:49 +01:00
Brian Gough
c32101f0fb upgrade redis-sharelatex to 1.0.10 2019-09-19 16:22:43 +01:00
Brian Gough
912a3a7753 remove redis server-side hashing for performance 
we still  compute the document hash in node, and check it on retrieval
but we don't check the hash at the point of writing it in redis which
was previously done with a redis Lua script.
 2019-09-09 15:27:58 +01:00
Henry Oswald
aa15a76059 added log lines for all project flushing 2019-08-30 07:38:53 +01:00
Henry Oswald
0ae838dd2d add logger into project flusher 2019-08-29 20:36:00 +01:00
Brian Gough
e75f2cc325 add unit tests for skipping history flush 2019-08-15 11:01:45 +01:00
Brian Gough
7493462154 add acceptance test for realtime shutdown 2019-08-15 10:38:07 +01:00
Brian Gough
a76e0dca88 skip history flush when project is cleared by realtime shutdown 
history is flushed by a background cron job anyway
 2019-08-15 09:51:16 +01:00
Simon Detheridge
f49f5cbf05 Merge pull request #79 from overleaf/ho-mongo-connection-string 
use MONGO_CONNECTION_STRING in settings.defaults.coffee if set
 2019-08-08 14:16:49 +01:00
Simon Detheridge
6721b904a7 Merge pull request #82 from overleaf/bg-mongo-health-check 
add a combined health check for mongo and redis
 2019-08-08 14:16:38 +01:00
Simon Detheridge
06444d2cc4 Improve/fix serializers for update logging (#80) 
* Improve/fix serializers for update logging
 2019-08-08 14:10:54 +01:00
Brian Gough
40f6494b19 add a combined health check for mongo and redis 2019-08-07 16:25:23 +01:00
Henry Oswald
aa1c74a979 use MONGO_CONNECTION_STRING in settings.defaults.coffee if set 2019-08-07 15:44:57 +01:00
Simon Detheridge
df9ca8b272 Add serializer to print only length of large fields in production 2019-07-31 16:42:28 +01:00
Brian Gough
618880f99d remove unnecessary check for doc_id 2019-07-24 16:57:43 +01:00
Brian Gough
c9ccf62d71 support per-doc pubsub channels 2019-07-22 12:20:06 +01:00
Brian Gough
97487a077e fix cluster/sentinel connection with real-time 2019-07-10 09:42:05 +01:00
Henry Oswald
06ad0f7acd Merge pull request #75 from overleaf/ho-pubsub-connection 
Remove real time redis connection and consolidate on pubsub
 2019-07-08 13:58:41 +01:00
Henry Oswald
374585f47d Merge pull request #72 from overleaf/sk-dep-upgrades-2019-06 
Update logger, metrics, and redis
 2019-07-08 13:58:28 +01:00
Henry Oswald
3b3b2da0f5 add pubsub redis connection and remove real time redis connection 2019-07-04 13:34:31 +01:00
Henry Oswald
eed8e215d7 bump redis-sharelatex (and io redis) to 1.0.9 2019-07-02 16:58:30 +01:00
Shane Kilkelly
e8a8c446ed Increase timeout on test? 2019-06-19 11:34:38 +01:00
Shane Kilkelly
080b482e51 Update logger, metrics, and redis 
Also fix acceptance tests, broken by a change in the redis driver
behaviour. It now returns promises from most operations, which confuses mocha.
 2019-06-19 10:25:54 +01:00
Henry Oswald
fdef197271 Merge branch 'master' into ho-detailed-flush-status 2019-06-13 14:33:22 +01:00
Henry Oswald
d9a737f97c return failed and succesfully flushed projects when flushing everything 2019-06-13 14:21:38 +01:00
Brian Gough
e8dd1aae9c Merge pull request #70 from overleaf/bg-metric-for-invalid-hash 
add metric for invalid hash and other sharejs errors
 2019-06-12 13:50:34 +01:00
Brian Gough
d50b93df2f add metric for invalid hash and other sharejs errors 2019-06-11 16:48:06 +01:00
Brian Gough
984b2d38e6 add unit tests 2019-06-11 16:33:14 +01:00
Brian Gough
966478cac4 fix hash check to use 'v' field instead of version 2019-06-11 14:11:46 +01:00
Brian Gough
e95059f98e handle non-urgent flushes in background 2019-06-03 10:01:10 +01:00
Brian Gough
0bbfa7de27 Merge branch 'master' into bg-downgrade-delete-component-error 2019-05-08 09:07:02 +01:00
Henry Oswald
d169cca425 Merge branch 'master' into ho-flush-all-projects 2019-05-07 17:02:14 +01:00
Brian Gough
27a8248196 convert "Delete component" errors into warnings 2019-05-07 16:55:17 +01:00
Tim Alby
8b40da701e update README 
- remove build status badge
- change app name
- update copyright notice
- update links
 2019-05-07 17:03:09 +02:00
Tim Alby
d316f172bf update repo URL for Jenkins 2019-05-07 17:03:06 +02:00
Henry Oswald
e57741cb80 stub out project flusher for unit tests 2019-05-07 15:46:30 +01:00
Henry Oswald
4b8a27a220 change github url 2019-05-07 14:01:12 +01:00
Tim Alby
ca6bfd8929 Update README 2019-05-06 17:28:52 +02:00
Henry Oswald
daca83a057 add dryRun option to flush all projects 2019-05-02 16:54:22 +01:00
Henry Oswald
d5d1736a5e adds /flush_all_projects project 2019-05-02 16:43:35 +01:00
Tim Alby
c1c23e4bee record last author id on document flush 
This is a multi-steps process:
* get a update's `user_id` from the metadata
* store the `user_id` (`lastUpdatedBy`) and current date (`lastUpdatedAt`) for
  the document in Redis on every updates
* fetch `lastUpdatedAt` and `lastUpdatedBy` from Redis on document flush
* send the data to web to be persisted in Mongo
 2019-05-02 11:10:02 +01:00
Brian Gough
68e7b9c4e9 Merge pull request #48 from sharelatex/bg-check-incoming-hash 
check incoming hash when present
 2019-04-29 10:15:44 +01:00
James Allen
52f3596e53 Review feedback 2019-04-16 11:05:17 +01:00
James Allen
33478f95fd Fix package.json versions 2019-04-11 16:32:31 +01:00
James Allen
3d76f4b9bf Record a snapshot to mongo when a doc's comments/changes get collapsed 2019-04-11 13:27:46 +01:00
Brian Gough
3c635c8d98 check version before it is modified by applyOp 2019-04-09 09:20:48 +01:00
Brian Gough
cc1f3fce5b check incoming hash when present 2019-04-08 14:12:18 +01:00
Brian Gough
fd1425d83f include a unique id in every message published to redis 2019-03-21 12:10:15 +00:00
Henry Oswald
4d7f9f3c1a add sentry into settings.defaults 2019-03-05 17:34:27 +00:00
Brian Gough
11f07d1f09 Merge pull request #45 from sharelatex/bg-avoid-text-html-content-type-in-responses 
use explicit json content-type to avoid security issues with text/html
 2019-02-22 09:24:30 +00:00
Henry Oswald
937e8fe071 use redis cluster key in defaults for pendingupdates 2019-02-15 13:58:29 +00:00
Brian Gough
8c5d74faef use explicit json content-type to avoid security issues with text/html 2019-02-12 16:45:11 +00:00
Henry Oswald
0a0fc91f28 Update app.coffee 2019-02-12 13:27:13 +00:00
Henry Oswald
3bc4cb492a added log line 2019-02-07 16:30:53 +00:00
Henry Oswald
08723f8972 revert health check redis types 2019-02-07 15:53:26 +00:00
Henry Oswald
ecaef6485b revert the removal of realtime keyspace 2019-02-07 15:27:51 +00:00
Henry Oswald
b5564095f3 add /health_check/redis route back in 2019-02-07 15:10:40 +00:00
Henry Oswald
73bd264401 remove realtime keys in settings, no longer used 2019-02-07 14:55:24 +00:00
Henry Oswald
2998750a33 fix redis version lock 2019-02-06 16:01:44 +00:00
Henry Oswald
4e1a2c787c Revert "turn down logging, use logger.info for less important data" 
This reverts commit c5f91428e3c7702fbbd3ffd1ef7a772d513f33f2.
 2019-02-06 15:29:22 +00:00
Henry Oswald
8d3fb729c5 point to web_api 2019-02-01 20:10:47 +00:00
Henry Oswald
e44a02391a standardise name of document-updater 2019-02-01 20:04:43 +00:00
Henry Oswald
98de529e42 make config look like chef config 2019-02-01 19:52:00 +00:00
Henry Oswald
3051d7db4b bulk upgrade to 1.1.12, moved to npm rather than git 2019-01-31 16:00:24 +00:00
Henry Oswald
d279a9149c add dispatcherCount to config file 2019-01-25 17:24:19 +00:00
Henry Oswald
76b054c7f4 bump node redis to 1.0.5 2019-01-25 17:01:59 +00:00
Christopher Hoskin
3eb2d27d4a Add **/*.map to .gitignore 2019-01-08 15:01:48 +00:00
Christopher Hoskin
5db1913cec Bump settings to v1.1.0 2019-01-08 15:00:26 +00:00
Christopher Hoskin
a783c6b3cc Bump logger to 1.5.9 2019-01-04 09:33:08 +00:00
Christopher Hoskin
05a80c7cba Add app.map to .gitignore 2019-01-04 09:31:00 +00:00