Commit graph

590 commits

Author SHA1 Message Date
Brian Gough
a2e63d009e fix migration phase check 2019-12-16 09:55:26 +00:00
Brian Gough
a85dffbcef fix acceptance tests 2019-12-16 09:27:00 +00:00
Brian Gough
d0c5eb5698 support migration of project history keys to separate redis instance 2019-12-13 16:38:41 +00:00
Brian Gough
ad19fee667 add setting so that double flush is the default
can be disabled to stop flushing to track-changes
2019-11-25 13:36:25 +00:00
Brian Gough
4f6583bbf2 fix getDocVersion and add tests 2019-11-25 13:28:36 +00:00
Brian Gough
68e12f4d2d add metrics for queue operations 2019-11-25 10:51:10 +00:00
Brian Gough
8b73bb9f13 Merge branch 'master' into bg-filter-track-changes-updates 2019-11-22 10:41:33 +00:00
Brian Gough
b7055eecee add metrics for history flushes 2019-11-22 09:14:32 +00:00
Brian Gough
65cf4cf7c7 make flush to track-changes failsafe 2019-11-21 14:58:35 +00:00
Brian Gough
dcd7649bad filter track-changes updates for projects using project-history 2019-11-19 10:02:56 +00:00
Brian Gough
08ee58abb8 remove console.log from tests 2019-11-15 09:34:13 +00:00
Brian Gough
82a72c9b73 fix missing bodyParser limit 2019-11-14 16:32:59 +00:00
dependabot[bot]
6871889bfb Bump request from 2.25.0 to 2.47.0
Bumps [request](https://github.com/request/request) from 2.25.0 to 2.47.0.
- [Release notes](https://github.com/request/request/releases)
- [Changelog](https://github.com/request/request/blob/master/CHANGELOG.md)
- [Commits](https://github.com/request/request/compare/v2.25.0...v2.47.0)

Signed-off-by: dependabot[bot] <support@github.com>
2019-11-05 11:38:25 +00:00
dependabot[bot]
ccc072e9da Bump express from 3.3.4 to 3.11.0
Bumps [express](https://github.com/expressjs/express) from 3.3.4 to 3.11.0.
- [Release notes](https://github.com/expressjs/express/releases)
- [Changelog](https://github.com/expressjs/express/blob/master/History.md)
- [Commits](https://github.com/expressjs/express/compare/3.3.4...3.11.0)

Signed-off-by: dependabot[bot] <support@github.com>
2019-10-30 13:38:50 +00:00
Nate Stemen
1d3a1d22be bump build script to 1.1.24 2019-10-25 12:50:45 -04:00
Nate Stemen
1620956e2e replace private link with public one 2019-10-25 12:50:33 -04:00
Brian Gough
a9428c9ece Merge pull request #96 from overleaf/bg-add-ioredis-maxretriesperrequest
add ioredis maxRetriesPerRequest
2019-10-16 08:55:36 +01:00
Brian Gough
21a9126847 set maxRetriesPerRequest from environment variable 2019-10-16 08:44:08 +01:00
Brian Gough
781e01b8b4 Merge pull request #95 from overleaf/bg-update-ioredis
update to ioredis 4.14.1
2019-10-14 09:21:39 +01:00
Brian Gough
5a62632cfd add ioredis option maxRetriesPerRequest: 0
see https://github.com/luin/ioredis/issues/965
2019-10-04 11:53:14 +01:00
Brian Gough
51a821c03c remove old unused ioredis keepalive option 2019-10-04 11:51:37 +01:00
Brian Gough
18ccd11224 update to ioredis 4.14.1 2019-10-04 11:32:08 +01:00
Henry Oswald
e9efd6b93b logs out when a redis event occurs 2019-10-03 11:10:31 +01:00
Brian Gough
d82b180b76 avoid project history queues building up with deferred flush 2019-10-03 04:05:24 +01:00
Brian Gough
c1454bc4ac Merge pull request #92 from overleaf/bg-flush-queue-prod-fixes
add continuous background flush
2019-10-02 13:11:00 +01:00
Brian Gough
0c14b7d2f8 add comment about background flush limit 2019-10-01 15:06:01 +01:00
Brian Gough
ae3ebf2db6 start background flush after http server has started 2019-10-01 15:02:37 +01:00
Brian Gough
2845b23b70 add smoothing of delete spikes 2019-10-01 15:01:53 +01:00
Brian Gough
2c22a60052 add random jitter to cutoff time 2019-10-01 15:01:20 +01:00
Brian Gough
a32495d2b4 make background flush more adaptive 2019-10-01 14:09:41 +01:00
Brian Gough
73b4262186 add continuous background flush 2019-09-30 16:05:53 +01:00
Brian Gough
33fadf51c1 fix getDocTimestamps for multiple docs 2019-09-30 13:50:25 +01:00
Brian Gough
89a90399fe Merge pull request #89 from overleaf/bg-queue-deletes
queue deletes for deferred processing
2019-09-30 11:14:08 +01:00
John Lees-Miller
c5a9105c33 Update config 2019-09-28 11:07:33 +01:00
Brian Gough
260923f291 keep flushQueuedProjects in the foreground 2019-09-27 10:46:24 +01:00
Brian Gough
7561e05660 check timestamps array length 2019-09-27 10:39:56 +01:00
Brian Gough
0f0682df43 allow flush to complete in acceptance test 2019-09-26 16:00:11 +01:00
Brian Gough
b7f3b848af remove unused dryRun option
Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>
2019-09-26 15:50:55 +01:00
Brian Gough
3bc176259b fix log line 2019-09-26 15:46:54 +01:00
Brian Gough
8cdc8c410a fix error logging 2019-09-26 15:46:45 +01:00
Brian Gough
fc62abfcfa run flush of queued projects in the background 2019-09-26 15:46:14 +01:00
Brian Gough
ba35c73cb6 add comment about ZPOPMIN 2019-09-26 15:18:10 +01:00
Brian Gough
a709a0adaa for simplicity keep the cutoff time the same while flushing the queue 2019-09-26 15:05:38 +01:00
Brian Gough
eae4b352ca remove unnecessary check 2019-09-26 14:59:03 +01:00
Brian Gough
9f358ead9f add an acceptance test for flush with queue processing 2019-09-26 10:55:05 +01:00
Brian Gough
b49621b3e9 add comments 2019-09-26 10:14:49 +01:00
Brian Gough
83dd43b809 add metric for queue length 2019-09-25 17:04:36 +01:00
Brian Gough
f6b2ac7360 queue deletes for deferred processing 2019-09-25 16:42:49 +01:00
Brian Gough
c32101f0fb upgrade redis-sharelatex to 1.0.10 2019-09-19 16:22:43 +01:00
Brian Gough
912a3a7753 remove redis server-side hashing for performance
we still  compute the document hash in node, and check it on retrieval
but we don't check the hash at the point of writing it in redis which
was previously done with a redis Lua script.
2019-09-09 15:27:58 +01:00