Brian Gough
fc62abfcfa
run flush of queued projects in the background
2019-09-26 15:46:14 +01:00
Brian Gough
ba35c73cb6
add comment about ZPOPMIN
2019-09-26 15:18:10 +01:00
Brian Gough
a709a0adaa
for simplicity keep the cutoff time the same while flushing the queue
2019-09-26 15:05:38 +01:00
Brian Gough
eae4b352ca
remove unnecessary check
2019-09-26 14:59:03 +01:00
Brian Gough
b49621b3e9
add comments
2019-09-26 10:14:49 +01:00
Brian Gough
83dd43b809
add metric for queue length
2019-09-25 17:04:36 +01:00
Brian Gough
f6b2ac7360
queue deletes for deferred processing
2019-09-25 16:42:49 +01:00
Brian Gough
912a3a7753
remove redis server-side hashing for performance
...
we still compute the document hash in node, and check it on retrieval
but we don't check the hash at the point of writing it in redis which
was previously done with a redis Lua script.
2019-09-09 15:27:58 +01:00
Henry Oswald
aa15a76059
added log lines for all project flushing
2019-08-30 07:38:53 +01:00
Henry Oswald
0ae838dd2d
add logger into project flusher
2019-08-29 20:36:00 +01:00
Brian Gough
a76e0dca88
skip history flush when project is cleared by realtime shutdown
...
history is flushed by a background cron job anyway
2019-08-15 09:51:16 +01:00
Simon Detheridge
6721b904a7
Merge pull request #82 from overleaf/bg-mongo-health-check
...
add a combined health check for mongo and redis
2019-08-08 14:16:38 +01:00
Simon Detheridge
06444d2cc4
Improve/fix serializers for update logging ( #80 )
...
* Improve/fix serializers for update logging
2019-08-08 14:10:54 +01:00
Brian Gough
40f6494b19
add a combined health check for mongo and redis
2019-08-07 16:25:23 +01:00
Simon Detheridge
df9ca8b272
Add serializer to print only length of large fields in production
2019-07-31 16:42:28 +01:00
Brian Gough
618880f99d
remove unnecessary check for doc_id
2019-07-24 16:57:43 +01:00
Brian Gough
c9ccf62d71
support per-doc pubsub channels
2019-07-22 12:20:06 +01:00
Brian Gough
97487a077e
fix cluster/sentinel connection with real-time
2019-07-10 09:42:05 +01:00
Henry Oswald
06ad0f7acd
Merge pull request #75 from overleaf/ho-pubsub-connection
...
Remove real time redis connection and consolidate on pubsub
2019-07-08 13:58:41 +01:00
Henry Oswald
3b3b2da0f5
add pubsub redis connection and remove real time redis connection
2019-07-04 13:34:31 +01:00
Brian Gough
16fb297043
Revert "skip hash check when non-BMP characters replaced"
2019-06-27 11:39:45 +01:00
Brian Gough
f37860599d
skip hash check when non-BMP characters replaced
2019-06-25 16:36:10 +01:00
Henry Oswald
fdef197271
Merge branch 'master' into ho-detailed-flush-status
2019-06-13 14:33:22 +01:00
Henry Oswald
d9a737f97c
return failed and succesfully flushed projects when flushing everything
2019-06-13 14:21:38 +01:00
Brian Gough
e8dd1aae9c
Merge pull request #70 from overleaf/bg-metric-for-invalid-hash
...
add metric for invalid hash and other sharejs errors
2019-06-12 13:50:34 +01:00
Brian Gough
d50b93df2f
add metric for invalid hash and other sharejs errors
2019-06-11 16:48:06 +01:00
Brian Gough
966478cac4
fix hash check to use 'v' field instead of version
2019-06-11 14:11:46 +01:00
Brian Gough
e95059f98e
handle non-urgent flushes in background
2019-06-03 10:01:10 +01:00
Brian Gough
0bbfa7de27
Merge branch 'master' into bg-downgrade-delete-component-error
2019-05-08 09:07:02 +01:00
Brian Gough
27a8248196
convert "Delete component" errors into warnings
2019-05-07 16:55:17 +01:00
Henry Oswald
daca83a057
add dryRun option to flush all projects
2019-05-02 16:54:22 +01:00
Henry Oswald
d5d1736a5e
adds /flush_all_projects project
2019-05-02 16:43:35 +01:00
Tim Alby
c1c23e4bee
record last author id on document flush
...
This is a multi-steps process:
* get a update's `user_id` from the metadata
* store the `user_id` (`lastUpdatedBy`) and current date (`lastUpdatedAt`) for
the document in Redis on every updates
* fetch `lastUpdatedAt` and `lastUpdatedBy` from Redis on document flush
* send the data to web to be persisted in Mongo
2019-05-02 11:10:02 +01:00
Brian Gough
68e7b9c4e9
Merge pull request #48 from sharelatex/bg-check-incoming-hash
...
check incoming hash when present
2019-04-29 10:15:44 +01:00
James Allen
52f3596e53
Review feedback
2019-04-16 11:05:17 +01:00
James Allen
3d76f4b9bf
Record a snapshot to mongo when a doc's comments/changes get collapsed
2019-04-11 13:27:46 +01:00
Brian Gough
3c635c8d98
check version before it is modified by applyOp
2019-04-09 09:20:48 +01:00
Brian Gough
cc1f3fce5b
check incoming hash when present
2019-04-08 14:12:18 +01:00
Brian Gough
fd1425d83f
include a unique id in every message published to redis
2019-03-21 12:10:15 +00:00
Brian Gough
8c5d74faef
use explicit json content-type to avoid security issues with text/html
2019-02-12 16:45:11 +00:00
Henry Oswald
3bc4cb492a
added log line
2019-02-07 16:30:53 +00:00
Henry Oswald
ecaef6485b
revert the removal of realtime keyspace
2019-02-07 15:27:51 +00:00
Henry Oswald
4e1a2c787c
Revert "turn down logging, use logger.info for less important data"
...
This reverts commit c5f91428e3c7702fbbd3ffd1ef7a772d513f33f2.
2019-02-06 15:29:22 +00:00
Christopher Hoskin
1217d8a80a
Merge branch 'master' into csh-ho-docker-issue-1338-bulk-upgrade
2019-01-04 09:18:40 +00:00
Henry Oswald
2505be283a
turn down logging, use logger.info for less important data
2018-12-13 12:46:10 +00:00
Henry Oswald
25e1ad7491
remove redis.realtime
...
real time pushes data into the document updater redis instance
2018-12-05 16:17:41 +00:00
Brian Gough
aae1352519
ensure that project history is flushed when the project is deleted
2018-10-30 12:03:44 +00:00
Brian Gough
537bd9a0c0
Merge pull request #36 from sharelatex/bg-limit-resync-requests-to-web
...
limit resync requests to web (connects to overleaf/sharelatex#775 )
2018-08-16 12:25:16 +01:00
Brian Gough
910b27357d
add more logging to resync operations
2018-08-16 11:14:11 +01:00
Brian Gough
aa013f0bee
limit parallel resync doc requests to web
2018-08-16 11:13:11 +01:00
