github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-10-31 21:21:03 -04:00
Code Issues Projects Releases Packages Wiki Activity
17766 commits 3 branches 7 tags 199 MiB
Commit graph

275 commits

Author SHA1 Message Date
Brian Gough
d0c5eb5698 support migration of project history keys to separate redis instance 2019-12-13 16:38:41 +00:00
Brian Gough
ad19fee667 add setting so that double flush is the default 
can be disabled to stop flushing to track-changes
 2019-11-25 13:36:25 +00:00
Brian Gough
4f6583bbf2 fix getDocVersion and add tests 2019-11-25 13:28:36 +00:00
Brian Gough
8b73bb9f13 Merge branch 'master' into bg-filter-track-changes-updates 2019-11-22 10:41:33 +00:00
Brian Gough
b7055eecee add metrics for history flushes 2019-11-22 09:14:32 +00:00
Brian Gough
dcd7649bad filter track-changes updates for projects using project-history 2019-11-19 10:02:56 +00:00
Brian Gough
f6b2ac7360 queue deletes for deferred processing 2019-09-25 16:42:49 +01:00
Brian Gough
912a3a7753 remove redis server-side hashing for performance 
we still  compute the document hash in node, and check it on retrieval
but we don't check the hash at the point of writing it in redis which
was previously done with a redis Lua script.
 2019-09-09 15:27:58 +01:00
Brian Gough
e75f2cc325 add unit tests for skipping history flush 2019-08-15 11:01:45 +01:00
Brian Gough
97487a077e fix cluster/sentinel connection with real-time 2019-07-10 09:42:05 +01:00
Henry Oswald
3b3b2da0f5 add pubsub redis connection and remove real time redis connection 2019-07-04 13:34:31 +01:00
Brian Gough
16fb297043 Revert "skip hash check when non-BMP characters replaced" 2019-06-27 11:39:45 +01:00
Brian Gough
f37860599d skip hash check when non-BMP characters replaced 2019-06-25 16:36:10 +01:00
Brian Gough
984b2d38e6 add unit tests 2019-06-11 16:33:14 +01:00
Brian Gough
e95059f98e handle non-urgent flushes in background 2019-06-03 10:01:10 +01:00
Brian Gough
0bbfa7de27 Merge branch 'master' into bg-downgrade-delete-component-error 2019-05-08 09:07:02 +01:00
Brian Gough
27a8248196 convert "Delete component" errors into warnings 2019-05-07 16:55:17 +01:00
Henry Oswald
e57741cb80 stub out project flusher for unit tests 2019-05-07 15:46:30 +01:00
Tim Alby
c1c23e4bee record last author id on document flush 
This is a multi-steps process:
* get a update's `user_id` from the metadata
* store the `user_id` (`lastUpdatedBy`) and current date (`lastUpdatedAt`) for
  the document in Redis on every updates
* fetch `lastUpdatedAt` and `lastUpdatedBy` from Redis on document flush
* send the data to web to be persisted in Mongo
 2019-05-02 11:10:02 +01:00
James Allen
52f3596e53 Review feedback 2019-04-16 11:05:17 +01:00
James Allen
3d76f4b9bf Record a snapshot to mongo when a doc's comments/changes get collapsed 2019-04-11 13:27:46 +01:00
Brian Gough
fd1425d83f include a unique id in every message published to redis 2019-03-21 12:10:15 +00:00
Brian Gough
8c5d74faef use explicit json content-type to avoid security issues with text/html 2019-02-12 16:45:11 +00:00
Henry Oswald
ecaef6485b revert the removal of realtime keyspace 2019-02-07 15:27:51 +00:00
Henry Oswald
4e1a2c787c Revert "turn down logging, use logger.info for less important data" 
This reverts commit c5f91428e3c7702fbbd3ffd1ef7a772d513f33f2.
 2019-02-06 15:29:22 +00:00
Christopher Hoskin
1217d8a80a Merge branch 'master' into csh-ho-docker-issue-1338-bulk-upgrade 2019-01-04 09:18:40 +00:00
Henry Oswald
2505be283a turn down logging, use logger.info for less important data 2018-12-13 12:46:10 +00:00
Henry Oswald
25e1ad7491 remove redis.realtime 
real time pushes data into the document updater redis instance
 2018-12-05 16:17:41 +00:00
Brian Gough
aae1352519 ensure that project history is flushed when the project is deleted 2018-10-30 12:03:44 +00:00
Brian Gough
fdcb806518 set a timestamp for the first entry in the projectHistory:Ops queue 2018-07-24 09:17:31 +01:00
Brian Gough
e471730efb Revert "Merge pull request #33 from sharelatex/bg-add-timestamp-marker-to-project-history-queue" 
This reverts commit 8ec398b133aa75c45384dd7ceb630cd743f4a15c, reversing
changes made to e3a6c934cf53fd245d7d0df737463cc90f954885.
 2018-07-23 16:05:04 +01:00
Brian Gough
6237577f87 set a timestamp for the first entry in the projectHistory:Ops queue 2018-07-20 10:43:31 +01:00
Hayden Faulds
62a10f25f2 move tk call to before sandboxed module call 2018-06-05 13:54:30 +01:00
Hayden Faulds
af92ca70a1 coerce projectHistoryId to integer after reading from Redis 2018-04-23 15:19:06 +01:00
Hayden Faulds
1b63141e49 Revert "Revert "Add projectHistoryId to updates"" 2018-04-23 12:08:04 +01:00
Hayden Faulds
3cc20d6311 Revert "Add projectHistoryId to updates" 2018-04-23 11:41:29 +01:00
Hayden Faulds
25c0be4eaa add projectHistryId to project structure updates 2018-04-13 15:22:05 +01:00
Hayden Faulds
a220794d32 add projectHistoryId to resync updates 2018-04-13 15:22:05 +01:00
Hayden Faulds
fb1852a593 cache projectHistoryId with doc in Redis 2018-04-13 15:22:05 +01:00
Brian Gough
3385d2640a fix structure ordering bug 2018-03-15 16:27:10 +00:00
Brian Gough
b3887fd984 update unit tests for incoming project versions 2018-03-15 16:24:11 +00:00
Brian Gough
0642e3c8c9 support project version on incoming requests 2018-03-15 16:15:10 +00:00
Hayden Faulds
4a5731440a Merge pull request #21 from sharelatex/hof-history-sync 
resync project history endpoint
 2018-03-14 10:40:59 +00:00
Hayden Faulds
1226f96fc3 extract ProjectHistoryRedisManager 2018-03-09 15:36:38 +00:00
Hayden Faulds
a72d228354 add unit tests 2018-03-07 13:29:53 +00:00
James Allen
939eaa2d4b Don't allow a document to be loaded without a pathname 2018-03-02 10:02:49 +00:00
Hayden Faulds
5a11332aa3 remove unecessary method wrapping 2018-02-22 10:17:59 +00:00
Hayden Faulds
ea0dd9700b reduce the number of times we flush project history 2018-02-22 10:01:05 +00:00
James Allen
3d5740fd7d Merge pull request #18 from sharelatex/msw-flush-project-ops 
Flush project structure changes every 500
 2018-02-16 10:30:00 +00:00
James Allen
657da70d45 Fix unit test namespacing from mocha upgrade 2018-02-15 16:28:58 +00:00
James Allen
f6c79f3203 Fix and extend unit tests 2018-02-15 15:55:12 +00:00
Michael Walker
241d1b27d5 Remove _shouldFlushHistoryOps wrapper 2018-01-31 11:27:40 +00:00
Michael Walker
3028fb9c3d Only flush project updates when crossing the threshold 2018-01-24 11:57:03 +00:00
Brian Gough
41f15c4fa8 fix unit test 2017-12-14 14:46:27 +00:00
Brian Gough
947e18eccf Merge branch 'bg-limit-ops-in-lock' 2017-12-14 14:26:44 +00:00
James Allen
6abf9c227d Calculate and send doc length before each update 2017-12-11 17:26:32 +00:00
Hayden Faulds
a3420b1236 version entity additions 2017-11-10 14:54:56 +00:00
Hayden Faulds
944e633bac add acceptance test for entity renaming 2017-11-08 09:20:38 +00:00
Hayden Faulds
7e86afe55e version file renames 2017-11-08 09:20:38 +00:00
Hayden Faulds
6d571e6d23 version document renames 2017-11-08 09:20:37 +00:00
Hayden Faulds
be41a1614d decorate ops with pathname and doc_length 2017-10-23 15:32:25 +01:00
Hayden Faulds
962520fca8 flush track-changes and project-history in HistoryManager 2017-10-23 15:32:25 +01:00
Hayden Faulds
d003aef31c conditionally enqueue history updates for project 2017-10-23 15:32:25 +01:00
Hayden Faulds
c27df0bfef split apart multi and rclient in tests 2017-10-23 14:24:23 +01:00
Hayden Faulds
81f998afe9 return pathname from DocumentManager.getDoc 2017-10-23 14:23:21 +01:00
Hayden Faulds
748315aadc handle pathname in DocumentManager.getDoc 2017-10-23 14:23:21 +01:00
Hayden Faulds
cbdace7386 store pathname in Redis cache 2017-10-23 14:23:21 +01:00
Hayden Faulds
05b93a629a return pathname from PersistenceManager 2017-10-23 14:17:14 +01:00
Brian Gough
02d3d1bd17 fix unit tests 
timekeeper must be called before SandboxedModule.require
 2017-10-23 14:02:53 +01:00
Brian Gough
7c2a28c161 fix use of timekeeper in unit tests 2017-10-23 13:33:14 +01:00
Brian Gough
7cbb3e7af8 change getProjectDocs endpoint from GET to POST 
also note that it flushes docs if they are considered old (i.e. not
recently flushed)
 2017-10-11 15:30:29 +01:00
Brian Gough
8d3f82360c update DocumentManager.getDoc signature 
keep alreadyLoaded flag at the end for consistency
 2017-10-11 14:25:33 +01:00
Brian Gough
2bbbf3c005 add unflushed time to doc in redis 2017-10-06 12:34:28 +01:00
Brian Gough
ad745f133d Merge pull request #59 from sharelatex/bg-reset-project-state 
add endpoint to clear project state
 2017-09-13 10:39:47 +01:00
Brian Gough
2047bd5516 fix broken unit test 2017-09-11 09:44:05 +01:00
Brian Gough
f34c12fdc7 add endpoint to clear project state 2017-09-08 15:50:26 +01:00
Brian Gough
9d931a3fa7 fix broken unit tests 2017-08-18 12:08:39 +01:00
Brian Gough
230c93766d fix broken RedisManager unit test 2017-08-18 11:59:31 +01:00
Brian Gough
f6be68eb08 added missing unit tests file 2017-08-11 16:49:16 +01:00
Brian Gough
f696ccb0d9 add unit tests for getProjectDocs 2017-08-10 16:41:28 +01:00
Brian Gough
ca7d9dce85 fix unit tests 2017-08-08 09:40:39 +01:00
Shane Kilkelly
c3465cf3d4 Reduce retry delay to 10ms, and refactor 2017-07-18 15:28:18 +01:00
Shane Kilkelly
0d46c3e2b3 WIP: auto-retry web requests 2017-07-17 10:08:21 +01:00
Brian Gough
d2730c8d6b unit tests for locking timeouts 2017-07-12 10:48:32 +01:00
Brian Gough
963e513057 add a timeout for getPreviousDocOps 
it uses several redis operations and this makes it prone to subsequent
timeouts if getDoc succeeds but is slow
 2017-06-23 15:50:21 +01:00
James Allen
7d22e4422c Merge pull request #49 from sharelatex/bg-add-redis-timeout 
add redis timeout
 2017-06-13 14:46:32 +01:00
James Allen
770b53f6af Add missing mocks to speed up tests and clean up output 2017-06-08 16:43:27 +01:00
Brian Gough
1ef258c878 add a timeout on the redis getDoc request 2017-06-07 11:34:42 +01:00
Brian Gough
729216c9b1 add unit tests for DocsIn check 2017-05-31 16:08:45 +01:00
Brian Gough
bed64d26c8 check if doc is missing from DocsIn set 2017-05-31 15:33:59 +01:00
Brian Gough
891ffda3bf put a rate limit on worker calls 2017-05-26 15:08:35 +01:00
Brian Gough
b15d2ef796 Revert "fail safely if doc cannot be loaded" 2017-05-15 16:18:40 +01:00
Paulo Jorge Reis
2c3f0aa6b4 Merge pull request #34 from sharelatex/pr-bulk-actions 
Add methods to bulk accept changes.
 2017-05-15 15:14:33 +01:00
Paulo Reis
05d7d1b8c6 Merge branch 'master' into pr-bulk-actions 2017-05-15 10:35:16 +01:00
Paulo Reis
36d16a76e0 Unit test accept changes in the ranges manager. 2017-05-12 14:42:40 +01:00
Brian Gough
2ee40d0748 ensure document is in redis before consuming ops 2017-05-12 14:22:18 +01:00
Brian Gough
d4a8d88750 put a limit on the number of ops per iteration 2017-05-12 13:15:59 +01:00
Paulo Reis
f690ef078a Update tests; add new ones for multiple changes. 2017-05-09 16:16:25 +01:00
Brian Gough
36407ac726 rename HistoryManager pushUncompressedHistoryOps 2017-05-09 12:02:27 +01:00
Brian Gough
fdf5e8e0b8 get history ops length directly from redis update 2017-05-09 10:34:31 +01:00