Jakob Ackermann
9cfc59734a
[misc] update the build scripts to 1.3.5
2020-02-11 12:27:53 +01:00
Brian Gough
2d2e3bc2b1
Merge pull request #111 from overleaf/bg-add-comment-about-null-bytes
...
add comment about null byte check
2020-02-10 10:26:24 +00:00
Brian Gough
afe43fa252
update to node 10.19.0
2020-02-07 14:16:31 +00:00
Jakob Ackermann
4070b0ef6f
[misc] tests: fix a static test
...
`ProjectHistoryRedisManager.queueOps` is masked and can not push
changes into redis during the unittest.
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2020-02-07 12:36:46 +01:00
Jakob Ackermann
10d177f41b
[misc] test/unit: stub the correct redis settings
2020-02-07 12:24:57 +01:00
Brian Gough
338d3609f5
add comment about null byte check
2020-01-30 15:17:13 +00:00
Brian Gough
544ae05212
added note about rollback
2020-01-23 16:22:26 +00:00
Brian Gough
626e19ed1a
add logging of migration phase at startup
2020-01-23 15:46:54 +00:00
Brian Gough
d5a2b96df9
add note about deleting the migration key entries
2020-01-23 14:36:59 +00:00
Jakob Ackermann
fcfa3ecc9d
[misc] upgrade node to 10.18.1
2020-01-23 10:05:57 +01:00
Brian Gough
31324fb65a
add environment variable for migration_phase setting
2020-01-22 15:36:06 +00:00
Brian Gough
ad58fe76b2
add tls settings
2020-01-21 15:36:33 +00:00
Brian Gough
531d9b77b9
add redislabs ca cert to repository
2020-01-21 09:49:41 +00:00
Brian Gough
7036803acf
add missing argument to metrics.inc
...
also track retries rather than attempts (which is always 1 for a successful request)
2020-01-14 15:00:21 +00:00
Brian Gough
3caa0e7c05
add failure/retry metrics for web-api requests
2020-01-14 13:53:50 +00:00
Brian Gough
a638ef4251
add comment about locking in redis migration
2020-01-13 15:56:28 +00:00
Brian Gough
c2714f9ae9
add acceptance tests for RedisMigrationManager
2020-01-06 16:50:06 +00:00
Brian Gough
27044c2d02
allow migration phase to be modified at runtime for testing
2020-01-06 16:46:35 +00:00
Brian Gough
8ae95ebf60
fix rclient check in migration metrics
2020-01-06 16:45:36 +00:00
Brian Gough
97cbf46160
add metrics for migration
2019-12-16 11:46:35 +00:00
Brian Gough
a2e63d009e
fix migration phase check
2019-12-16 09:55:26 +00:00
Brian Gough
a85dffbcef
fix acceptance tests
2019-12-16 09:27:00 +00:00
Brian Gough
d0c5eb5698
support migration of project history keys to separate redis instance
2019-12-13 16:38:41 +00:00
Brian Gough
1bb12a5035
allow pending updates to clear in acceptance tests
2019-12-11 14:43:59 +00:00
Brian Gough
ad19fee667
add setting so that double flush is the default
...
can be disabled to stop flushing to track-changes
2019-11-25 13:36:25 +00:00
Brian Gough
4f6583bbf2
fix getDocVersion and add tests
2019-11-25 13:28:36 +00:00
Brian Gough
68e12f4d2d
add metrics for queue operations
2019-11-25 10:51:10 +00:00
Brian Gough
8b73bb9f13
Merge branch 'master' into bg-filter-track-changes-updates
2019-11-22 10:41:33 +00:00
Brian Gough
b7055eecee
add metrics for history flushes
2019-11-22 09:14:32 +00:00
Brian Gough
65cf4cf7c7
make flush to track-changes failsafe
2019-11-21 14:58:35 +00:00
Brian Gough
dcd7649bad
filter track-changes updates for projects using project-history
2019-11-19 10:02:56 +00:00
Brian Gough
08ee58abb8
remove console.log from tests
2019-11-15 09:34:13 +00:00
Brian Gough
82a72c9b73
fix missing bodyParser limit
2019-11-14 16:32:59 +00:00
dependabot[bot]
6871889bfb
Bump request from 2.25.0 to 2.47.0
...
Bumps [request](https://github.com/request/request ) from 2.25.0 to 2.47.0.
- [Release notes](https://github.com/request/request/releases )
- [Changelog](https://github.com/request/request/blob/master/CHANGELOG.md )
- [Commits](https://github.com/request/request/compare/v2.25.0...v2.47.0 )
Signed-off-by: dependabot[bot] <support@github.com>
2019-11-05 11:38:25 +00:00
dependabot[bot]
ccc072e9da
Bump express from 3.3.4 to 3.11.0
...
Bumps [express](https://github.com/expressjs/express ) from 3.3.4 to 3.11.0.
- [Release notes](https://github.com/expressjs/express/releases )
- [Changelog](https://github.com/expressjs/express/blob/master/History.md )
- [Commits](https://github.com/expressjs/express/compare/3.3.4...3.11.0 )
Signed-off-by: dependabot[bot] <support@github.com>
2019-10-30 13:38:50 +00:00
Nate Stemen
1d3a1d22be
bump build script to 1.1.24
2019-10-25 12:50:45 -04:00
Nate Stemen
1620956e2e
replace private link with public one
2019-10-25 12:50:33 -04:00
Brian Gough
a9428c9ece
Merge pull request #96 from overleaf/bg-add-ioredis-maxretriesperrequest
...
add ioredis maxRetriesPerRequest
2019-10-16 08:55:36 +01:00
Brian Gough
21a9126847
set maxRetriesPerRequest from environment variable
2019-10-16 08:44:08 +01:00
Brian Gough
781e01b8b4
Merge pull request #95 from overleaf/bg-update-ioredis
...
update to ioredis 4.14.1
2019-10-14 09:21:39 +01:00
Brian Gough
5a62632cfd
add ioredis option maxRetriesPerRequest: 0
...
see https://github.com/luin/ioredis/issues/965
2019-10-04 11:53:14 +01:00
Brian Gough
51a821c03c
remove old unused ioredis keepalive option
2019-10-04 11:51:37 +01:00
Brian Gough
18ccd11224
update to ioredis 4.14.1
2019-10-04 11:32:08 +01:00
Henry Oswald
e9efd6b93b
logs out when a redis event occurs
2019-10-03 11:10:31 +01:00
Brian Gough
d82b180b76
avoid project history queues building up with deferred flush
2019-10-03 04:05:24 +01:00
Brian Gough
c1454bc4ac
Merge pull request #92 from overleaf/bg-flush-queue-prod-fixes
...
add continuous background flush
2019-10-02 13:11:00 +01:00
Brian Gough
0c14b7d2f8
add comment about background flush limit
2019-10-01 15:06:01 +01:00
Brian Gough
ae3ebf2db6
start background flush after http server has started
2019-10-01 15:02:37 +01:00
Brian Gough
2845b23b70
add smoothing of delete spikes
2019-10-01 15:01:53 +01:00
Brian Gough
2c22a60052
add random jitter to cutoff time
2019-10-01 15:01:20 +01:00
Brian Gough
a32495d2b4
make background flush more adaptive
2019-10-01 14:09:41 +01:00
Brian Gough
73b4262186
add continuous background flush
2019-09-30 16:05:53 +01:00
Brian Gough
33fadf51c1
fix getDocTimestamps for multiple docs
2019-09-30 13:50:25 +01:00
Brian Gough
89a90399fe
Merge pull request #89 from overleaf/bg-queue-deletes
...
queue deletes for deferred processing
2019-09-30 11:14:08 +01:00
John Lees-Miller
c5a9105c33
Update config
2019-09-28 11:07:33 +01:00
Brian Gough
260923f291
keep flushQueuedProjects in the foreground
2019-09-27 10:46:24 +01:00
Brian Gough
7561e05660
check timestamps array length
2019-09-27 10:39:56 +01:00
Brian Gough
0f0682df43
allow flush to complete in acceptance test
2019-09-26 16:00:11 +01:00
Brian Gough
b7f3b848af
remove unused dryRun option
...
Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>
2019-09-26 15:50:55 +01:00
Brian Gough
3bc176259b
fix log line
2019-09-26 15:46:54 +01:00
Brian Gough
8cdc8c410a
fix error logging
2019-09-26 15:46:45 +01:00
Brian Gough
fc62abfcfa
run flush of queued projects in the background
2019-09-26 15:46:14 +01:00
Brian Gough
ba35c73cb6
add comment about ZPOPMIN
2019-09-26 15:18:10 +01:00
Brian Gough
a709a0adaa
for simplicity keep the cutoff time the same while flushing the queue
2019-09-26 15:05:38 +01:00
Brian Gough
eae4b352ca
remove unnecessary check
2019-09-26 14:59:03 +01:00
Brian Gough
9f358ead9f
add an acceptance test for flush with queue processing
2019-09-26 10:55:05 +01:00
Brian Gough
b49621b3e9
add comments
2019-09-26 10:14:49 +01:00
Brian Gough
83dd43b809
add metric for queue length
2019-09-25 17:04:36 +01:00
Brian Gough
f6b2ac7360
queue deletes for deferred processing
2019-09-25 16:42:49 +01:00
Brian Gough
c32101f0fb
upgrade redis-sharelatex to 1.0.10
2019-09-19 16:22:43 +01:00
Brian Gough
912a3a7753
remove redis server-side hashing for performance
...
we still compute the document hash in node, and check it on retrieval
but we don't check the hash at the point of writing it in redis which
was previously done with a redis Lua script.
2019-09-09 15:27:58 +01:00
Henry Oswald
aa15a76059
added log lines for all project flushing
2019-08-30 07:38:53 +01:00
Henry Oswald
0ae838dd2d
add logger into project flusher
2019-08-29 20:36:00 +01:00
Brian Gough
e75f2cc325
add unit tests for skipping history flush
2019-08-15 11:01:45 +01:00
Brian Gough
7493462154
add acceptance test for realtime shutdown
2019-08-15 10:38:07 +01:00
Brian Gough
a76e0dca88
skip history flush when project is cleared by realtime shutdown
...
history is flushed by a background cron job anyway
2019-08-15 09:51:16 +01:00
Simon Detheridge
f49f5cbf05
Merge pull request #79 from overleaf/ho-mongo-connection-string
...
use MONGO_CONNECTION_STRING in settings.defaults.coffee if set
2019-08-08 14:16:49 +01:00
Simon Detheridge
6721b904a7
Merge pull request #82 from overleaf/bg-mongo-health-check
...
add a combined health check for mongo and redis
2019-08-08 14:16:38 +01:00
Simon Detheridge
06444d2cc4
Improve/fix serializers for update logging ( #80 )
...
* Improve/fix serializers for update logging
2019-08-08 14:10:54 +01:00
Brian Gough
40f6494b19
add a combined health check for mongo and redis
2019-08-07 16:25:23 +01:00
Henry Oswald
aa1c74a979
use MONGO_CONNECTION_STRING in settings.defaults.coffee if set
2019-08-07 15:44:57 +01:00
Simon Detheridge
df9ca8b272
Add serializer to print only length of large fields in production
2019-07-31 16:42:28 +01:00
Brian Gough
618880f99d
remove unnecessary check for doc_id
2019-07-24 16:57:43 +01:00
Brian Gough
c9ccf62d71
support per-doc pubsub channels
2019-07-22 12:20:06 +01:00
Brian Gough
97487a077e
fix cluster/sentinel connection with real-time
2019-07-10 09:42:05 +01:00
Henry Oswald
06ad0f7acd
Merge pull request #75 from overleaf/ho-pubsub-connection
...
Remove real time redis connection and consolidate on pubsub
2019-07-08 13:58:41 +01:00
Henry Oswald
374585f47d
Merge pull request #72 from overleaf/sk-dep-upgrades-2019-06
...
Update logger, metrics, and redis
2019-07-08 13:58:28 +01:00
Henry Oswald
3b3b2da0f5
add pubsub redis connection and remove real time redis connection
2019-07-04 13:34:31 +01:00
Henry Oswald
eed8e215d7
bump redis-sharelatex (and io redis) to 1.0.9
2019-07-02 16:58:30 +01:00
Shane Kilkelly
e8a8c446ed
Increase timeout on test?
2019-06-19 11:34:38 +01:00
Shane Kilkelly
080b482e51
Update logger, metrics, and redis
...
Also fix acceptance tests, broken by a change in the redis driver
behaviour. It now returns promises from most operations, which confuses mocha.
2019-06-19 10:25:54 +01:00
Henry Oswald
fdef197271
Merge branch 'master' into ho-detailed-flush-status
2019-06-13 14:33:22 +01:00
Henry Oswald
d9a737f97c
return failed and succesfully flushed projects when flushing everything
2019-06-13 14:21:38 +01:00
Brian Gough
e8dd1aae9c
Merge pull request #70 from overleaf/bg-metric-for-invalid-hash
...
add metric for invalid hash and other sharejs errors
2019-06-12 13:50:34 +01:00
Brian Gough
d50b93df2f
add metric for invalid hash and other sharejs errors
2019-06-11 16:48:06 +01:00
Brian Gough
984b2d38e6
add unit tests
2019-06-11 16:33:14 +01:00
Brian Gough
966478cac4
fix hash check to use 'v' field instead of version
2019-06-11 14:11:46 +01:00
Brian Gough
e95059f98e
handle non-urgent flushes in background
2019-06-03 10:01:10 +01:00
Brian Gough
0bbfa7de27
Merge branch 'master' into bg-downgrade-delete-component-error
2019-05-08 09:07:02 +01:00
Henry Oswald
d169cca425
Merge branch 'master' into ho-flush-all-projects
2019-05-07 17:02:14 +01:00