github/overleaf
1
0
Fork 0
mirror of https://github.com/overleaf/overleaf.git synced 2024-11-21 20:47:08 -05:00
Code Issues Projects Releases Packages Wiki Activity
23671 commits 3 branches 7 tags 206 MiB
Commit graph

670 commits

Author SHA1 Message Date
Henry Oswald
31e1808dd8 shard the pending-updates-list queue 2021-02-08 16:02:41 +00:00
Henry Oswald
c308e60b6b speed up DocumentUpdaterControllerTests by directly injecting redis into 
tests
 2021-02-08 11:13:22 +00:00
Jakob Ackermann
c2f8f26da4 [misc] bump the node version to 10.23.1 2021-01-05 18:26:52 +00:00
John Lees-Miller
8fec6723d8 Add correct license 2020-12-03 10:27:38 +00:00
Brian Gough
1ddcdf38ca add comment about health checks on '/' 2020-11-27 14:42:42 +00:00
Jakob Ackermann
8a5858f430 Merge pull request #200 from overleaf/jpa-bump-metrics-module 
[misc] bump metrics module to 3.4.1
 2020-11-26 13:13:50 +00:00
Jakob Ackermann
746c5aeb80 [misc] bump metrics module to 3.4.1 
- renamed package from `metrics-sharelatex` to `@overleaf/metrics`
- drop support for statsd backend
- decaffeinate
- compress `/metrics` response using gzip
- bump debugging agents to latest versions
- expose prometheus interfaces for custom metrics (custom tags)
- cleanup of open sockets metrics
- fix deprecation warnings for header access
 2020-11-25 11:57:22 +00:00
Eric Mc Sween
96126ecf34 Upgrade build-scripts to 3.4.0 
This version fixes docker-compose health checks for dependent services. See
https://github.com/overleaf/dev-environment/pull/409 for details.
 2020-11-24 08:03:29 -05:00
Jakob Ackermann
15af5c7977 [misc] bump @overleaf/redis-wrapper to version 2.0.0 2020-11-11 16:24:22 +00:00
Jakob Ackermann
dea6968fa9 [misc] explicitly install underscore 
Use the same (outdated) version that is already in place.
 2020-11-11 16:23:12 +00:00
Jakob Ackermann
222caeb9a8 Merge pull request #196 from overleaf/jpa-drop-duplicate-join 
[RoomManager] drop duplicate joining of entities
 2020-10-19 17:01:40 +02:00
Jakob Ackermann
d6ac8c14e7 [RoomManager] drop duplicate joining of entities 
REF: 0437e1d03f89a058f97a8884e3532a9a58b68b9d
REF: 62be5e29e5232150e7063bc189c5ad8a1189f972
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
 2020-10-19 15:54:37 +01:00
Jakob Ackermann
b20cdc0122 [misc] bump the dev-env to 3.3.5 -- drop custom unit test timeout 2020-10-19 15:20:57 +01:00
Jakob Ackermann
e8911babe0 Merge pull request #182 from overleaf/dependabot-npm_and_yarn-lodash-4.17.20 
Bump lodash from 4.17.15 to 4.17.20
 2020-10-08 10:42:21 +02:00
Jakob Ackermann
e9c23b6409 Merge pull request #191 from overleaf/dependabot-npm_and_yarn-node-fetch-2.6.1 
Bump node-fetch from 2.6.0 to 2.6.1
 2020-10-08 10:42:08 +02:00
Jakob Ackermann
5b53832bf2 Merge pull request #192 from overleaf/jpa-bump-dev-env-3-3-4-node-10-22-1 
[misc] bump the dev-env to 3.3.4 and bump the node version to 10.22.1
 2020-10-08 10:41:56 +02:00
Jakob Ackermann
e846192db0 [MatrixTests] add a large testing matrix 
Layers/Dimensions:
- users: anonymous, registered, registeredWithOwnedProject
- session setup: noop, joinReadWriteProject, joinReadWriteProjectAndDoc,
   joinOwnProject, joinOwnProjectAndDoc
- invalid requests: noop, joinProjectWithDocId, joinDocWithDocId,
   joinProjectWithProjectId, joinDocWithProjectId,
   joinProjectWithProjectIdThenJoinDocWithDocId
 2020-10-07 16:20:21 +01:00
Jakob Ackermann
4ec73acb88 Merge branches 'jpa-prepare-for-matrix-tests' and 'jpa-setup-session-helper' into jpa-matrix-tests 2020-10-01 12:57:40 +01:00
Jakob Ackermann
2900c60c4a [FixturesManager] add a helper for setting up project and doc together 2020-10-01 12:33:15 +01:00
Jakob Ackermann
78fbd04ef8 [MockWebServer] grant users at least the privileges of anonymous users 2020-10-01 12:28:39 +01:00
Jakob Ackermann
4f860995d8 [MockDocUpdaterServer] return a 404 when a requested doc does not exist 2020-10-01 12:25:41 +01:00
Jakob Ackermann
cc681a94f4 [app] ignore error from writing to disconnected long-polling client 2020-09-22 14:10:23 +01:00
Jakob Ackermann
f3c7619fb5 [misc] revert back to high timeout for unit tests 2020-09-17 15:34:03 +01:00
Jakob Ackermann
ba3b4f7dcd [misc] bump the dev-env to 3.3.4 and bump the node version to 10.22.1 2020-09-17 15:26:45 +01:00
dependabot[bot]
ee06705377 Bump lodash from 4.17.15 to 4.17.20 
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.15 to 4.17.20.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.15...4.17.20)

Signed-off-by: dependabot[bot] <support@github.com>
 2020-09-16 10:38:21 +00:00
dependabot[bot]
10c6e01bd8 Bump node-fetch from 2.6.0 to 2.6.1 
Bumps [node-fetch](https://github.com/bitinn/node-fetch) from 2.6.0 to 2.6.1.
- [Release notes](https://github.com/bitinn/node-fetch/releases)
- [Changelog](https://github.com/node-fetch/node-fetch/blob/master/docs/CHANGELOG.md)
- [Commits](https://github.com/bitinn/node-fetch/compare/v2.6.0...v2.6.1)

Signed-off-by: dependabot[bot] <support@github.com>
 2020-09-12 15:08:05 +00:00
Jakob Ackermann
a2f5bf463e Merge pull request #190 from overleaf/jpa-socket-io-logger 
[misc] socket.io: use a custom logger
 2020-09-02 10:46:23 +02:00
Jakob Ackermann
c6f2a3b387 Merge pull request #189 from overleaf/jpa-relevel-log 
[misc] re-level expected error log messages
 2020-09-02 10:46:08 +02:00
Jakob Ackermann
72282a07df [misc] socket.io: use a custom logger 
- forward the previously enabled log messages to our logger-module
- stub the previously disabled logger methods
- drop the log-level config for socket.io
 2020-09-01 09:30:11 +01:00
Jakob Ackermann
4960bdd6fe [misc] re-level log: 404 from web -> WARN and emit 'project not found' 
A stale browser tab tried to join a deleted project.
Emitting 'project not found'/'ProjectNotFound' will trigger a page
 reload in the frontend, upon web can render a 404.
See frontend: ConnectionManager.joinProject callback
 2020-08-27 11:51:57 +01:00
Jakob Ackermann
55d938ba14 [misc] add tests for web replying with a 404 for joinProject 2020-08-27 11:51:56 +01:00
Jakob Ackermann
884b340c75 [misc] re-level log: 403 from web goes to WARN and emit 'not authorized' 
Users will get redirected to the login page and will see a 'restricted'
 page after logging in again.
See frontend: ConnectionManager.reportConnectionError
 2020-08-27 11:51:56 +01:00
Jakob Ackermann
d2a2b9d46e [misc] add tests for web replying with a 403 for joinProject 2020-08-27 11:51:56 +01:00
Jakob Ackermann
1ff9c1e71b [misc] add the rpc-method into the log context in Router._handleError 2020-08-27 11:51:55 +01:00
Jakob Ackermann
0647abf433 [misc] drop info-log in WebApiManager for joinProject being rate-limited 
The CodedError is logged at warn-level in Router._handleError.
 2020-08-27 11:51:09 +01:00
Jakob Ackermann
2ce7b36c95 [misc] drop duplicate log line for unauthorized applyOtUpdate calls 
The violation is logged in Router._handleError.
 2020-08-27 10:22:31 +01:00
Jakob Ackermann
dee4749e6d [misc] re-level log: properly silence unauthorized updateClientPosition 2020-08-27 10:11:40 +01:00
Jakob Ackermann
425052ff91 Merge pull request #187 from overleaf/jpa-o-error-tagging 
[misc] migrate to OError tagging/wrapping
 2020-08-25 11:46:28 +02:00
Jakob Ackermann
64e659bf43 Merge pull request #186 from overleaf/jpa-fix-join-project-error-context 
[misc] fix join project error context
 2020-08-25 11:46:18 +02:00
Jakob Ackermann
849a1cf416 Merge pull request #185 from overleaf/jpa-doc-id-in-error-context 
[misc] add/bring back doc_id in error context
 2020-08-25 11:42:41 +02:00
Jakob Ackermann
ee3d3b09ed [misc] wrap redis errors as tagging does not work with them 
ioredis may reuse the error instance for multiple callbacks. E.g. when
 the connection to redis fails, the queue is flushed with the same
 MaxRetriesPerRequestError instance.
 2020-08-24 10:12:20 +01:00
Jakob Ackermann
537e97be73 [misc] OError.tag all the errors in async contexts 
See the docs of OError.tag:
https://github.com/overleaf/o-error#long-stack-traces-with-oerrortag
(currently at 221dd902e7bfa0ee92de1ea5a3cbf3152c3ceeb4)

I am tagging all errors at each async hop. Most of the controller code
 will only ever see already tagged errors -- or new errors created in
 our app code. They should have enough info that we do not need to tag
 them again.
 2020-08-24 10:12:06 +01:00
Jakob Ackermann
8e31cc5c23 [Router] _handleError: joinProject error-context may not have project_id 
The ol_context patch changed the priority of client context and rpc
 context.
This lead to the (possibly missing) project_id of the client context
 overwriting the project_id of the rpc context.
REF: f1d55c0a5437a518e9f4617473caed9ba928e648
 2020-08-21 13:29:28 +01:00
Jakob Ackermann
f935b1881a [Router] leaveDoc: pass the doc_id into the error-context 2020-08-21 12:47:42 +01:00
Jakob Ackermann
fd88819eec [Router] _handleError: ol_context.doc_id does not exist, drop overwrite 2020-08-21 12:47:42 +01:00
Jakob Ackermann
880056d397 [Router] use a new UnexpectedArgumentsError 2020-08-21 12:47:08 +01:00
Jakob Ackermann
50140f785a [WebsocketController] use a new JoinLeaveEpochMismatchError 2020-08-21 12:47:08 +01:00
Jakob Ackermann
0462e3e437 [WebsocketController] use a new NotJoinedError 2020-08-21 12:47:07 +01:00
Jakob Ackermann
4cb8cc4a85 [DocumentUpdaterManager] use a new ClientRequestedMissingOpsError 2020-08-21 12:47:07 +01:00
Jakob Ackermann
8abfdb87ff [DocumentUpdaterManager] use a new DocumentUpdaterRequestFailedError 2020-08-21 12:47:07 +01:00
Jakob Ackermann
02a2382264 [WebApiManager] use a new CorruptedJoinProjectResponseError 2020-08-21 12:47:07 +01:00
Jakob Ackermann
68bc9d0d23 [WebApiManager] use a new WebApiRequestFailedError 2020-08-21 12:47:06 +01:00
Jakob Ackermann
59c4c884a5 [WebsocketController] use the new NotAuthorizedError 2020-08-21 12:47:06 +01:00
Jakob Ackermann
a8c51de510 [AuthorizationManager] use a new NotAuthorizedError 2020-08-21 12:47:06 +01:00
Jakob Ackermann
de518ea4eb [SessionSockets] use a new MissingSessionError 2020-08-21 12:47:05 +01:00
Jakob Ackermann
6828becb46 [DocumentUpdaterManager] use a new NullBytesInOpError 2020-08-21 12:47:05 +01:00
Jakob Ackermann
af50f9b02c [DocumentUpdaterManager] use a new UpdateTooLargeError 2020-08-21 12:47:05 +01:00
Jakob Ackermann
5950b26a42 [SafeJsonParse] migrate to OError and use a new DataTooLargeToParseError 2020-08-21 12:47:05 +01:00
Jakob Ackermann
f82177a46a [Errors] migrate to OError 2020-08-21 12:47:04 +01:00
Jakob Ackermann
c7b9dadd16 [misc] increase the timeout for unit tests -- CI boxes are too slow 2020-08-21 12:45:30 +01:00
Jakob Ackermann
7ce76775ea Merge pull request #181 from overleaf/jpa-forcefully-disconnect-from-shutdown-process 
[misc] forcefully disconnect from shutdown process
 2020-08-17 12:56:18 +02:00
Jakob Ackermann
1ffffbcbc2 Merge pull request #180 from overleaf/jpa-harden-cleanup-take2 
[misc] bump socket.io and socket.io-client
 2020-08-17 12:56:04 +02:00
Jakob Ackermann
67d9543cf3 Merge pull request #179 from overleaf/csh-issue-1123-bump-logger-220 
[misc] bump logger-sharelatex to version 2.2.0
 2020-08-17 12:55:40 +02:00
Jakob Ackermann
3505fae253 Merge pull request #178 from overleaf/jpa-ignore-broken-socket-errors 
[app] ignore errors from accessing disconnected client sockets
 2020-08-17 12:55:29 +02:00
Jakob Ackermann
1550e9f885 Merge pull request #176 from overleaf/jpa-bump-dev-env-3-3-2 
[misc] bump the dev-env to 3.3.2
 2020-08-17 12:55:13 +02:00
Jakob Ackermann
83b8b077fa [misc] bump socket.io and socket.io-client 2020-08-14 11:16:26 +01:00
Jakob Ackermann
ea75b84eef [misc] let the orchestrator handle the process restart 
Note that there is also the `shutdownCleanly` interval which may notice
 that the shutdown has completed. There is some network IO required to
 signal all clients the server disconnect, so we cannot run process.exit
 immediately.
 2020-08-13 13:39:25 +01:00
Jakob Ackermann
ee59056c6e [misc] forcefully disconnect stale clients from shutdown process 2020-08-13 13:39:22 +01:00
Christopher Hoskin
fe41f1b00c [misc] bump logger-sharelatex to version 2.2.0 2020-08-12 16:04:21 +01:00
Jakob Ackermann
a0028646a5 [app] ignore errors from accessing disconnected client sockets 
Technically `EHOSTUNREACH` and `ETIMEDOUT` should go into a
 'disconnected_read' metric. But this would require changes to
 dashboards and a larger diff.
 2020-08-12 11:45:49 +01:00
Brian Gough
831d794bf4 clean up join/leave handling 
Co-Authored-By: Jakob Ackermann <jakob.ackermann@overleaf.com>
 2020-08-12 10:54:22 +01:00
Jakob Ackermann
370baa06c3 [misc] bump the dev-env to 3.3.2 2020-08-10 17:10:31 +01:00
Jakob Ackermann
51e56dafcf [misc] bump the dev-env to 3.3.1 2020-08-05 10:46:23 +01:00
Jakob Ackermann
1b52366c61 Merge pull request #174 from overleaf/jpa-bump-dev-env 
[misc] bump the dev-env to 3.3.0
 2020-08-04 12:29:22 +02:00
Jakob Ackermann
f4d9ae1150 [misc] bump the dev-env to 3.3.0 2020-08-04 11:07:19 +01:00
Jakob Ackermann
a91f78634b [misc] bump socket.io and socket.io-client 2020-07-31 16:32:03 +01:00
Shane Kilkelly
3d92858939 Upgrade redis-sharelatex to 1.0.13 2020-07-24 10:42:46 +01:00
Jakob Ackermann
562375d351 [misc] fix express deprecations 2020-07-22 09:45:14 +01:00
Jakob Ackermann
ca4168ce90 [misc] use overleaf/socket.io and overleaf/socket.io-client forks 2020-07-08 15:07:19 +01:00
Jakob Ackermann
9fff03bca5 [misc] optionally expose the hostname in the 'debug.getHostname' rpc 2020-07-07 11:07:28 +01:00
Jakob Ackermann
aa9d6c8dc9 [misc] reland decaff cleanup (#166) 
* [misc] decaff cleanup: RoomManager

* [misc] decaff cleanup: RedisClientManager

* [misc] decaff cleanup: SafeJsonParse

* [misc] decaff cleanup: WebApiManager

* [misc] decaff cleanup: WebsocketController

* [misc] decaff cleanup: WebsocketLoadBalancer

* [misc] decaff cleanup: SessionSockets

* [misc] decaff cleanup: HttpController

* [misc] decaff cleanup: HttpApiController

* [misc] decaff cleanup: HealthCheckManager

* [misc] decaff cleanup: EventLogger

* [misc] decaff cleanup: Errors

o-error will eliminate most of it -- when we migrate over.

* [misc] decaff cleanup: DrainManager

* [misc] decaff cleanup: DocumentUpdaterManager

* [misc] decaff cleanup: DocumentUpdaterController: no-unused-vars

* [misc] decaff cleanup: DocumentUpdaterController: Array.from

* [misc] decaff cleanup: DocumentUpdaterController: implicit return

* [misc] decaff cleanup: DocumentUpdaterController: IIFE

* [misc] decaff cleanup: DocumentUpdaterController: null checks

* [misc] decaff cleanup: DocumentUpdaterController: simpler loops

* [misc] decaff cleanup: DocumentUpdaterController: move module name def

* [misc] decaff cleanup: ConnectedUsersManager: handle-callback-err

* [misc] decaff cleanup: ConnectedUsersManager: implicit returns

* [misc] decaff cleanup: ConnectedUsersManager: null checks

* [misc] decaff cleanup: ChannelManager: no-unused-vars

* [misc] decaff cleanup: ChannelManager: implicit returns

* [misc] decaff cleanup: ChannelManager: other cleanup

- var -> const
- drop variable assignment before return

* [misc] decaff cleanup: AuthorizationManager: handle-callback-err

Note: This requires a change in WebsocketController to provide a dummy
 callback.

* [misc] decaff cleanup: AuthorizationManager: Array.from

* [misc] decaff cleanup: AuthorizationManager: implicit returns

* [misc] decaff cleanup: AuthorizationManager: null checks

* [misc] decaff cleanup: Router: handle-callback-err

* [misc] decaff cleanup: Router: standard/no-callback-literal

* [misc] decaff cleanup: Router: Array.from

* [misc] decaff cleanup: Router: implicit returns

* [misc] decaff cleanup: Router: refactor __guard__ wrapper

* [misc] decaff cleanup: Router: null checks

And a minor bug fix: user.id -> user._id

* [misc] decaff cleanup: Router: move variable declarations to assignments

* [misc] decaff cleanup: app: implicit returns

* [misc] decaff cleanup: app: __guard__

* [misc] decaff cleanup: app: null checks

* [misc] decaff cleanup: app: function definitions

* [misc] decaff cleanup: app: drop unused next argument

* [misc] decaff cleanup: app: var -> const
 2020-07-07 11:06:02 +01:00
Jakob Ackermann
8601084a2e [misc] WebsocketControllerTests: remove debugging code 2020-07-01 11:36:36 +01:00
Jakob Ackermann
a77222470b [misc] WebsocketControllerTests: always pass cb to updateClientPosition 
...and fix the inconsistent async behaviour for logged-in vs anonymous
 users in the app.
 2020-06-29 18:11:33 +01:00
Jakob Ackermann
4d01c02946 [misc] WebsocketControllerTests: explicitly pass a cb to leaveProject 
...and fix the stubbing of `io.sockets.clients`.

We were running the assertions prior to the actual completion of the
 leaveProject. Which in turn shadowed the broken test setup --
 `io.sockets.clients()` was returning the `this.clientsInRoom=[]` from a
 previous context.
 2020-06-29 17:38:22 +01:00
Jakob Ackermann
bea0e9beb7 [misc] RoomManagerTests: explicitly pass a cb to joinProject/joinDoc 2020-06-29 17:12:26 +01:00
Jakob Ackermann
5e191be171 [misc] replace console logging with logger-sharelatex 2020-06-24 11:12:22 +01:00
Jakob Ackermann
fa42166be3 [misc] ignore camelcase warning in settings (key_schema parameter) 2020-06-24 10:52:38 +01:00
Jakob Ackermann
9c7bfe020c [misc] fix eslint errors in app.js 2020-06-24 10:52:38 +01:00
Jakob Ackermann
c76bcb7732 [misc] fix eslint errors in XMLHttpRequest.js 2020-06-24 10:52:37 +01:00
Jakob Ackermann
b8b3fb8b11 [misc] fix usage of deprecated node apis 2020-06-24 10:52:21 +01:00
decaffeinate
53058452ee prettier: convert miscellaneous files to Prettier format 2020-06-24 10:28:28 +01:00
decaffeinate
92dede867f prettier: convert individual decaffeinated files to Prettier format 2020-06-23 18:30:51 +01:00
decaffeinate
bdfca5f155 decaffeinate: convert individual files to js 2020-06-23 18:30:48 +01:00
decaffeinate
42f55c4651 decaffeinate: rename individual coffee files to js files 2020-06-23 18:30:46 +01:00
decaffeinate
8a7fc78dc8 prettier: convert test/acceptance decaffeinated files to Prettier format 2020-06-23 18:30:45 +01:00
decaffeinate
db98fdac0a decaffeinate: rename test/acceptance/coffee to test/acceptance/js 2020-06-23 18:30:37 +01:00
decaffeinate
5443450abb decaffeinate: Run post-processing cleanups on ApplyUpdateTests.coffee and 18 other files 2020-06-23 18:30:34 +01:00
decaffeinate
d318e4fd0e decaffeinate: Convert ApplyUpdateTests.coffee and 18 other files to JS 2020-06-23 18:30:29 +01:00
decaffeinate
30a9c6ed2c decaffeinate: Rename ApplyUpdateTests.coffee and 18 other files from .coffee to .js 2020-06-23 18:30:23 +01:00
decaffeinate
3eceb8a5f6 prettier: convert test/unit decaffeinated files to Prettier format 2020-06-23 18:30:16 +01:00
decaffeinate
68e2adebf5 decaffeinate: rename test/unit/coffee to test/unit/js 2020-06-23 18:30:06 +01:00
decaffeinate
93697a8c5c decaffeinate: Run post-processing cleanups on AuthorizationManagerTests.coffee and 13 other files 2020-06-23 18:30:03 +01:00
decaffeinate
2ca620e7a0 decaffeinate: Convert AuthorizationManagerTests.coffee and 13 other files to JS 2020-06-23 18:29:59 +01:00
decaffeinate
e5d07bd3af decaffeinate: Rename AuthorizationManagerTests.coffee and 13 other files from .coffee to .js 2020-06-23 18:29:53 +01:00
decaffeinate
817844515d prettier: convert app/js decaffeinated files to Prettier format 2020-06-23 18:29:44 +01:00
decaffeinate
04a85a6716 decaffeinate: rename app/coffee dir to app/js 2020-06-23 18:29:41 +01:00
decaffeinate
a397154e18 decaffeinate: Run post-processing cleanups on AuthorizationManager.coffee and 18 other files 2020-06-23 18:29:38 +01:00
decaffeinate
7335084c26 decaffeinate: Convert AuthorizationManager.coffee and 18 other files to JS 2020-06-23 18:29:34 +01:00
decaffeinate
90eafa388a decaffeinate: Rename AuthorizationManager.coffee and 18 other files from .coffee to .js 2020-06-23 18:29:29 +01:00
decaffeinate
20bb3540e7 decaffeinate: update .gitignore 2020-06-23 18:29:21 +01:00
decaffeinate
59083edb9e decaffeinate: update build scripts to es 2020-06-23 18:29:17 +01:00
Jakob Ackermann
8ac973e767 [misc] help prettier/decaffeinate with scoping a comment into a fn body 2020-06-23 18:04:09 +01:00
Jakob Ackermann
205efa8812 [WebsocketController] reset the client context before (re)populating it 
Co-Authored-By: Brian Gough <brian.gough@overleaf.com>
 2020-06-22 17:04:31 +01:00
Jakob Ackermann
1fcf534dcf [Router] revert preserving of client.ol_context 
We do not enter this line twice, it would result in multiple event
 handlers too.
 2020-06-22 17:04:31 +01:00
Jakob Ackermann
d17ef183d0 [Router] gracefully set and do not reset the ol_context 2020-06-22 17:04:30 +01:00
Jakob Ackermann
ce4f9148c3 [perf] WebsocketLoadBalancer: move back to a sync loop for msg fan out 2020-06-22 17:04:15 +01:00
Jakob Ackermann
a70c1e1fa2 [perf] WebsocketLoadBalancer: check is_restricted_message once 
...and filter clients early on.
 2020-06-22 17:04:15 +01:00
Jakob Ackermann
5282f8f531 [misc] synchronous client store using an Object at .ol_context 2020-06-22 17:04:08 +01:00
Miguel Serrano
f2cae26166 Merge pull request #157 from overleaf/msm-update-deps 
Update Dependencies
 2020-06-22 15:45:11 +02:00
Miguel Serrano
3ffc218b2c Rolled back redis-sharelatex to 1.0.12 2020-06-22 13:13:44 +02:00
Miguel Serrano
ce3266c59c Fixed sha downgrades 2020-06-19 17:29:31 +02:00
Miguel Serrano
1e1f6ca19f Updated 'uid-safe' and minor/patch dependencies 2020-06-19 16:26:32 +02:00
Jakob Ackermann
5f7841526f [misc] RoomManager: emitOnCompletion: properly handle Promise rejections 
```
result = Promise.all([<Promise that rejects eventually>]) # rejection 1
result.then () -> RoomEvents.emit(eventName)              # rejection 2
result.catch (err) -> RoomEvents.emit(eventName, err)     # handle r1
```
As shown above, the second rejection remains unhandled. The fix is to
 chain the `.catch()` onto the `.then()` Promise.
 2020-06-17 09:29:12 +01:00
Jakob Ackermann
06a8dc8ec1 Merge pull request #155 from overleaf/jpa-backport-receiveupdate-tests 
[backport] 107 and 116: Improve ReceiveUpdate acceptance tests
 2020-06-10 11:28:25 +02:00
Jakob Ackermann
10eb30e572 Merge pull request #154 from overleaf/jpa-backport-httpcontroller-tests 
[backport] 111 HttpController: 404 for missing clients and add tests
 2020-06-10 11:28:17 +02:00
Jakob Ackermann
45ac2bdd97 Merge pull request #153 from overleaf/jpa-backport-drainmanager-tests 
[backport] 108 and 112: DrainManager acceptance tests
 2020-06-10 11:28:08 +02:00
Jakob Ackermann
cfe37dcbb5 Merge pull request #152 from overleaf/jpa-vendor-cookie 
[misc] vendor a patched session.socket.io middleware
 2020-06-10 11:27:58 +02:00
Jakob Ackermann
5b85e0154d Merge pull request #151 from overleaf/jpa-perf-parser 
[perf] add a few short cuts to the packet decoding
 2020-06-10 11:27:50 +02:00
Jakob Ackermann
b2e4448992 [misc] test/acceptance: ReceiveUpdateTests: test remotely sent update 2020-06-10 09:53:48 +01:00
Jakob Ackermann
83e3ff0ed7 [misc] test/acceptance: ReceiveUpdateTests: add 2nd project/3rd client 
...and check for cross project leakage.
 2020-06-10 09:53:29 +01:00
Jakob Ackermann
56fda1f9b0 [misc] test/acceptance: use the correct redis instances 2020-06-10 09:43:15 +01:00
Jakob Ackermann
bc44494466 [HttpController] return 404 in case of a missing client and add tests 
Add acceptance tests for the client view.
 2020-06-10 09:41:36 +01:00
Jakob Ackermann
eabff1d6b2 [perf] test/acceptance: DrainManagerTests: cleanup previous clients 
...before starting to drain.
 2020-06-09 18:02:38 +01:00
Jakob Ackermann
de35fc5ecf [HttpApiController] implement the disconnection of a single client 
The http route returns as soon as the client has fully disconnected.
 2020-06-09 18:01:08 +01:00
Jakob Ackermann
91e296533f [misc] test/acceptance: add tests for the draining of connections 2020-06-09 18:00:44 +01:00
Jakob Ackermann
acb7d7df5a [misc] add test cases for the validation of the callback argument 
When the user provides a function as last argument for socket.emit,
 socket.io will flag this as an RPC and add a cb as the last argument
 to the client.on('event', ...) handler on the server side.
Without a function as last argument for socket.emit, the callback
 argument on the server side is undefined, leading to invalid function
 calls (`undefined()`) and an unhandled exception.
The user can also provide lots of other arguments, so the 2nd/3rd ...
 argument is of arbitrary type, again leading to invalid function calls
 -- e.g. `1()`.
 2020-06-09 16:30:03 +01:00
Jakob Ackermann
853ee994a6 [perf] add a few short cuts to the packet decoding 2020-06-09 15:26:17 +01:00
Jakob Ackermann
dc553c4150 [misc] vendor a patched session.socket.io middleware 2020-06-09 15:21:33 +01:00
Jakob Ackermann
1c9eaf574a Merge pull request #147 from overleaf/jpa-backport-141 
[backport] 141: Router: validate the callback argument
 2020-06-09 14:41:24 +02:00
Jakob Ackermann
5238e6ad36 Merge pull request #145 from overleaf/jpa-backport-140 
[backport] 140: stop processing requests as we detect a client disconnect
 2020-06-09 14:41:12 +02:00
Jakob Ackermann
589bedc3dd Merge pull request #146 from overleaf/jpa-downgrade-logging 
[misc] downgrade logging when running tests
 2020-06-09 14:40:59 +02:00
Jakob Ackermann
32af7001fc [misc] Router: prefix the publicId with 'P.' for easy differentiation 2020-06-08 11:29:40 +01:00
Jakob Ackermann
f40241a037 [misc] downgrade logging when running tests 2020-06-05 11:38:09 +01:00
Jakob Ackermann
0b2cccf1e0 [misc] apply review feedback: adjust metric names 
Co-Authored-By: Brian Gough <brian.gough@overleaf.com>

(cherry-picked from commit 67674b83efb452ece05cdc39525ee3a5eeb8a4d7)
 2020-06-05 11:33:00 +01:00
Jakob Ackermann
ddcb9cf8c8 [misc] downgrade a warning message from clients leaving non-joined rooms 
This can now happen all the time, as we skip the join for clients that
 disconnect before joinProject/joinDoc completed.

(cherry-picked from commit f357931de74e088800f3cced3898cce4f251dad0)
 2020-06-05 11:32:43 +01:00
Jakob Ackermann
7fa9061015 [misc] stop processing requests as we detect a client disconnect 
v2 expose `client.connected`; v0 exposes `client.disconnected`

(cherry-picked from commit a9d70484343ca9be367d45bf7bb949e4be449647)
 2020-06-05 11:32:13 +01:00
Jakob Ackermann
0840700ffd [Router] validate the callback argument 2020-06-05 10:59:01 +01:00
Jakob Ackermann
c6d08647c7 [misc] socket.io: use a secondary publicId for public facing usages 2020-06-04 17:18:07 +01:00
Brian Gough
f973b377f0 update node to v10.21.0 2020-06-03 09:12:21 +01:00
Tim Alby
94d57f50c0 add fake lint and format targets 
Highly hacky!
Lint and format steps are coming very soon thanks to the decaf, but in the
meantime we need steps to pass CI. Updating the build scripts after the
decaf will undo this change.
 2020-05-28 16:09:27 +02:00
Jakob Ackermann
d13acb8ca3 [ChannelManager] port related and still mostly valid test from v2 
I skipped the bulk of verifyConsistentBehaviour tests which are not
 valid for the new implementation -- there is no optimization and
 always cleanup.
 2020-05-15 18:39:33 +02:00
Jakob Ackermann
41debfae0f [ChannelManager] rework (un)subscribing to redis 
- send a subscribe request on every request
- wait for a pending unsubscribe request before subscribing
- wait for a pending subscribe request before unsubscribing

Co-Authored-By: Brian Gough <brian.gough@overleaf.com>
 2020-05-15 18:34:33 +02:00
Jakob Ackermann
1095851dfe [misc] test/unit: fix typos and assertion of error messages 
Sinon does not check the contents of the passed error when checked via
 sinon.stub().calledWith.
```
callback = sinon.stub()
callback(new Error("some message"))
  .calledWith(new Error("completely different message"))
  === true
```

Cherry-pick plus an additional patch for the joinProject bail-out.
(cherry picked from commit d9570fee70701a5f431c39fdbec5f8bc5a7843fe)
 2020-05-15 14:46:05 +02:00
Jakob Ackermann
17d04b9041 [misc] bump sinon to 2.x for promise support with stubs 
(cherry picked from commit 3c92b937f9430175d7c51660e03c507635448e88)
 2020-05-15 12:01:22 +02:00
Jakob Ackermann
b713beb7f0 Merge pull request #135 from overleaf/jpa-skip-leave-project-for-invalid-clients 
[WebsocketController] skip leaveProject when joinProject didn't complete
 2020-05-13 15:34:41 +02:00
Jakob Ackermann
55af5e502f [WebsocketController] skip leaveProject when joinProject didn't complete 
Also drop dead code:
 - user_id bailout

   There is a check on a completed joinProject call now. It will always
    set a user_id, see Router.coffee which has a fallback `{_id:"..."}`.

 - late project_id bailout

   WebsocketLoadBalancer.emitToRoom will not work without a project_id.
   We have to bail out before the call.
 2020-05-12 17:15:08 +02:00
Jakob Ackermann
684cb3903c [WebsocketController] handle redis subscribe error on joinProject 
joinProject should not complete when the redis pub/sub subscribe request
 failed.
 2020-05-12 13:03:50 +02:00
Brian Gough
5c28da1031 add metric for pendingUpdates queue 2020-04-07 11:53:54 +01:00
Brian Gough
5765884f38 Merge branch 'master' into jpa-pub-sub-metrics 2020-04-07 11:46:23 +01:00
Henry Oswald
8711abdb66 bump redis to 1.0.12 2020-03-31 14:04:33 +01:00
Jakob Ackermann
56628a16c6 [misc] track redis pub/sub payload sizes on publish 2020-03-30 11:31:44 +02:00
Jakob Ackermann
a9b8e9be3b [misc] upgrade metrics-sharelatex to 2.6.2 2020-03-30 10:47:01 +02:00
Jakob Ackermann
69569e3571 [misc] config: add headroom for JSON serialization in maxUpdateSize 2020-03-24 16:21:29 +01:00
Jakob Ackermann
af53d3b603 [misc] skip duplicate JSON serialization for size check 2020-03-24 11:22:28 +01:00
Jakob Ackermann
cb675d38c2 [misc] SafeJsonParse: align the size limit with the frontend->rt limit 
frontend -> real-time and doc-updater -> real-time should be in sync.
Otherwise we can send a payload to doc-updater, but can not receive the
 confirmation of it -- and the client will send it again in a loop.

Also log the size of the payload.
 2020-03-24 09:14:15 +01:00
Jakob Ackermann
15244a54be [misc] WebsocketController: limit the update size to 7mb 
bail out early on -- especially do not push the update into redis for
 doc-updater to discard it.

Confirm the update silently, otherwise the frontend will send it again.
Broadcast a 'otUpdateError' message and disconnect the client, like
 doc-updater would do.
 2020-03-24 09:12:12 +01:00
Jakob Ackermann
d320c2d5f3 [misc] let proxys observe an upcoming shutdown before starting to drain 
Otherwise clients may be routed to the same pod upon reconnecting.
 2020-03-17 16:41:48 +01:00
Jakob Ackermann
2b1c5bf436 Merge pull request #100 from overleaf/jpa-dependencies-cleanup 
[misc] cleanup dependencies
 2020-02-14 10:56:11 +01:00
Jakob Ackermann
43013e0820 [misc] cleanup unused dependency on mongo 2020-02-12 14:44:01 +01:00
Jakob Ackermann
902b4fca46 [misc] rename npm-shrinkwrap.json to package-lock.json and run npm i 2020-02-12 14:39:53 +01:00
Brian Gough
1ad8315437 remove unused .travis.yml file 2020-02-12 12:37:00 +00:00
Jakob Ackermann
24d46e9d4b [misc] update the build scripts to 1.3.5 2020-02-11 12:27:56 +01:00
Brian Gough
9a4124ee11 Merge pull request #97 from overleaf/bg-update-node 
Update to node:10.19.0 docker image
 2020-02-10 10:00:43 +00:00
Brian Gough
98b29889bd Merge pull request #95 from overleaf/bg-revert-unnecessary-changes 
Revert debugging PRs for node 10 upgrade
 2020-02-10 10:00:17 +00:00
Brian Gough
e0e2090a42 update node version in nvmrc and buildscripts 2020-02-07 14:41:12 +00:00
Brian Gough
04a9d66784 use public node:10.19.0 image 2020-02-07 14:15:48 +00:00
Brian Gough
abe4d1d525 update to gcr.io/overleaf-ops/node:10.19.0 2020-02-06 03:34:30 +00:00
Brian Gough
64bd739a87 Revert "Merge pull request #91 from overleaf/spd-trycatch-all-the-things" 
This reverts commit 2bf7f14f9d050c58f141f465633bb6e274b903dd, reversing
changes made to 989240812532ca43a52513339f4dda8f44a80a64.
 2020-02-05 10:05:36 +00:00
Brian Gough
4ec82b1baa upgrade to local node:10.18.1 image 2020-02-04 15:02:15 +00:00
Simon Detheridge
4102aa0580 Add more detail to metric 2020-02-04 14:03:56 +00:00
Simon Detheridge
8e45a62e32 Handle ECONNRESET in the same way as EPIPE 2020-02-04 13:58:45 +00:00
Simon Detheridge
7663e9689e Merge pull request #91 from overleaf/spd-trycatch-all-the-things 
Add try/catch around all client emissions
 2020-02-04 13:21:49 +00:00
Simon Detheridge
fbff3fe727 Don't shut down on uncaught EPIPE 2020-02-04 12:56:43 +00:00
Simon Detheridge
9fd4699098 Merge remote-tracking branch 'origin/bg-avoid-emit-when-client-not-connected' into spd-trycatch-all-the-things 2020-02-04 12:35:35 +00:00
Simon Detheridge
216a977922 Add try/catch around all client emissions 2020-02-04 12:13:03 +00:00
Brian Gough
ebb83e4633 use diconnected property, not connected 2020-02-04 11:59:25 +00:00
Brian Gough
7380d523d5 avoid emitting when client not connected 
the emit is happening asynchronously after the client list is computed,
so clients may have disconnected in the intervening time.
 2020-02-04 11:39:37 +00:00
Brian Gough
e263d37476 pass the signal correctly to the shutdown handler 2020-02-04 11:14:53 +00:00
Brian Gough
1fc8cc44c3 log shutdown messages as warnings 2020-02-04 11:14:14 +00:00
Simon Detheridge
c7e2b99a7b Update hybi-16 patch to work with socket.io 0.9.19 2020-02-04 10:43:06 +00:00
Simon Detheridge
ef852dfa33 Update socket.io to latest patch release 2020-02-04 10:32:54 +00:00
Brian Gough
4f94110225 Merge pull request #84 from overleaf/spd-catch-errors 
Bump to Node 10 and add error handlers for socket.io
 2020-02-03 14:59:04 +00:00
Brian Gough
49a8e1214b use a separate field for client errors 2020-02-03 14:47:45 +00:00
Brian Gough
e04b6e1e49 Update app/coffee/Router.coffee 
Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>
 2020-02-03 14:46:14 +00:00
Shane Kilkelly
e63c6f4395 Merge pull request #87 from overleaf/sk-restricted-users 
Filter "comments" if restricted user.
 2019-10-31 10:21:11 +00:00
Shane Kilkelly
6df88ebc49 Filter "comments" if restricted user. 2019-10-30 13:54:40 +00:00
Shane Kilkelly
403caa65e8 Revert "Revert "Track the isRestrictedUser flag on clients"" 
This reverts commit 651e392a7c644403f199e1b03e7494b61ce71d0c.
 2019-10-30 13:52:36 +00:00
Nate Stemen
9a838bd071 bump build script to 1.1.24 2019-10-25 13:23:13 -04:00
Nate Stemen
3dc7c357a5 add public link to contributing docs 2019-10-25 13:22:58 -04:00
Simon Detheridge
925a8651c1 Revert "Track the isRestrictedUser flag on clients" 2019-10-22 10:17:38 +01:00