Implement Lua scripts for atomic sequence incrementing and cold-start initialization in Redis to prevent race conditions. Improve the sequence buffer manager with CAS-like logic to prevent interval leakage during concurrent allocations. Update the message repository to use `SELECT ... FOR UPDATE` and conditional updates for `last_seq` to ensure database consistency during high concurrency.
- Add `nextSeqLua` and `initSeqLua` for atomic Redis operations
- Implement thread-safe buffer management in `seq_buffer.go`
- Add row-level locking in `message_repo.go`
- Update service layers to support group-aware sequence retrieval
Update the `Seq` field in `WSEventResponse` and its usages across message and websocket handlers to use `int64` instead of `string`. This aligns the data transfer object with the underlying message sequence type and removes unnecessary string conversions.
Introduce several performance and synchronization enhancements:
- Implement `SeqBufferManager` to allow sequence number pre-allocation via Redis Lua scripts, reducing atomic increment overhead.
- Add `PushWorker` to handle asynchronous message pushing using Redis Streams.
- Implement incremental conversation synchronization via `ConversationVersionLog` to allow clients to fetch only recent changes.
- Add support for Gzip compression in WebSocket communications to reduce bandwidth usage.
- Update dependency injection and configuration to support these new components.
Improve the reliability of conversation sequence generation by implementing a fallback mechanism to the database when Redis is unavailable or when keys are missing.
- Add `GetNextSeq` to `MessageRepository` and implement it in `MessageRepositoryAdapter`.
- Update `ConversationCache` to fallback to DB in `GetConvMaxSeq`, `GetConvMaxSeqBatch`, and `GetNextSeq`.
- Ensure Redis TTL is refreshed after every `INCR` operation to prevent sequence counter expiration.
- Refactor service layers to handle sequence generation errors more explicitly.
- Add `IsEnabled` helper to `LayeredCache` for cleaner availability checks.
This commit performs a significant cleanup of the codebase by removing unused functions, methods, and entire files across various internal modules. This reduces technical debt and simplifies the project structure.
Key changes include:
- **cache**: Removed unused cache key generators and metrics snapshots.
- **dto**: Removed redundant converter functions and segment creation helpers.
- **middleware**: Deleted the unused `logger.go` middleware and simplified `ratelimit.go` and `casbin.go`.
- **model**: Removed unused ID helpers, database closing functions, and batch decryption logic.
- **pkg**: Cleaned up unused utility functions in `circuitbreaker`, `crypto`, `cursor`, `hook`, and `utils`.
- **service**: Deleted `account_cleanup_service.go` and removed unused helper functions in `log_cleanup_service.go` and `sensitive_service.go`.
- **repository**: Removed unused private loading methods in `comment_repo.go`.
Update the chat message push mechanism to include group ID and group avatar
when sending messages in group conversations. This ensures that push
notifications for group chats display the group's identity and icon
rather than the sender's personal avatar.
- Update `PushChatMessage` interface and implementation to accept `groupID` and `groupAvatar`.
- Modify `chat_service.go` to extract group metadata and pass it to the push service.
- Update `push_service.go` to include group details in the push payload extras and set the notification icon to the group avatar for group conversations.
Refactor unread count logic to use arithmetic calculation (maxSeq - readSeq)
instead of manual incrementing/decrementing. This simplifies the cache
management and improves consistency.
New features:
- Batch mark-as-read functionality for multiple conversations.
- Lightweight sync data endpoint to retrieve conversation metadata
(maxSeq and last message timestamp) for client synchronization.
- Optimized batch retrieval of conversation max sequences using Redis MGet.
Technical changes:
- Deprecated `IncrementUnread` and `ClearUnread` in `ConversationCache`.
- Added `GetConvMaxSeqBatch` to reduce network round-trips.
- Added `HandleMarkReadAll` and `HandleGetSyncData` handlers.
- Updated `ChatService` to support batch operations and sync data retrieval.
Add new functionality for managing and synchronizing academic grades and exam schedules. This includes new models, repositories, services, and HTTP handlers, along with updated gRPC definitions and dependency injection configuration.
Additionally, improve the reliability of unread message count tracking by implementing an atomic Lua script for Redis operations in the conversation cache.
Introduce a new mechanism for tracking read positions using message
sequences (hasReadSeq), similar to OpenIM, to allow for O(1) unread
count calculations.
- Implement `UserReadSeq` caching in Redis with Lua scripts to prevent
sequence regression (ensuring updates only occur if the new sequence
is greater than the current one).
- Update `ConversationCache` to support sequence-based unread count
computation (`maxSeq - hasReadSeq`).
- Refactor `MessageRepository` to use conditional updates for
`last_read_seq` in the database.
- Update `ChatService` to automatically update the sender's read
sequence when sending a message.
- Optimize `MarkAsRead` logic to update both database and Redis
sequence caches and refine notification targets for private vs
group chats.
- Update `jpush` client to use pointer types for boolean fields to
properly handle optional values in JSON.
Introduce a distributed task execution system for gRPC runners. This includes:
- A `TaskBus` for cluster-mode task dispatching via Redis Pub/Sub.
- A `RunnerRegistry` to track active runner instances in Redis.
- Support for both standalone and cluster modes via configuration.
- Refactored `RunnerHub` and `TaskManager` to use a `TaskDispatcher` interface, decoupling task submission from local execution.
- Added distributed locking to `HotRankWorker` to ensure only one instance performs periodic hot rank recalculations in a multi-instance deployment.
- Updated dependency injection (Wire) to support the new runner components and registry.
Introduce a new WebSocket messaging architecture that supports both standalone and cluster modes. This allows for horizontal scaling of WebSocket servers by using Redis Pub/Sub to synchronize messages across multiple instances.
Key changes:
- Added `ws.MessagePublisher` interface to abstract message distribution.
- Implemented `ws.Bus` to handle cluster-mode messaging via Redis.
- Added `ws.OnlineTracker` to manage user online status across the cluster.
- Refactored multiple services (Chat, Group, Push, Call, etc.) to use the new `MessagePublisher` instead of a concrete `ws.Hub`.
- Added WebSocket configuration options (mode, instance ID, channel, TTL, heartbeat) to `config.yaml` and `config.go`.
- Updated dependency injection with Wire to support the new publisher and Redis client.
- Improved logging by replacing standard `log` with `zap` in several service components.
Allow image segments with URLs matching the local S3 prefix to bypass
SSRF validation, as internal S3 endpoints may resolve to private IP
addresses that would otherwise trigger security errors.
This commit introduces several significant architectural improvements to enhance system performance, scalability, and reliability:
- **Performance Optimization (N+1 Problem Resolution)**: Implemented batching mechanisms across `MessageHandler`, `ChatService`, `GroupService`, `MessageService`, and `UserService`. This replaces multiple individual database/cache queries with single batch operations for fetching unread counts, last messages, participants, and user information.
- **Enhanced Unread Count Management**: Migrated unread count tracking to a Redis Hash-based approach (`unread#️⃣{userID}`). This allows for atomic increments/decrements and efficient retrieval of both individual conversation unread counts and total unread counts for a user.
- **Improved Message Sequencing**: Integrated Redis-based sequence (`seq`) pre-allocation for messages to ensure strict ordering and reduce database contention during high-concurrency message creation.
- **Push Service Reliability**: Refactored the push notification system to include persistent `PushRecord` tracking. Added a recovery mechanism (`recoverPendingPushes`) to reload pending notifications from the database upon service startup, ensuring better delivery guarantees.
- **WebSocket Reliability**: Updated the WebSocket hub to move away from in-memory history replay in favor of a client-driven synchronization model (`sync_required` event and `ack` handling), reducing memory overhead and improving connection stability.
- **Cache Layer Enhancements**: Added `HIncrBy` and `IncrBySeq` to the `Cache` interface and its implementations (`RedisCache`, `LayeredCache`) to support the new unread and sequence management logic.
This commit introduces several architectural improvements and optimizations across the codebase:
- **Performance & Reliability**:
- Implemented Redis pipelining in `ConversationCache.CacheMessage` to reduce network round-trips.
- Added a circuit breaker to the JPush client to prevent cascading failures.
- Introduced batch deletion and batch member addition capabilities in repositories.
- Added message idempotency support using `client_msg_id` and a Redis-based cache.
- Optimized WebSocket handling with connection limits (total and per-user) and improved error logging.
- **Code Refactoring**:
- Refactored `Router` to use a `RouterDeps` struct, simplifying the constructor and improving maintainability.
- Unified model ID generation logic using new `id_helper.go` (supporting UUID and Snowflake).
- Standardized JSON serialization/deserialization in models using `json_helper.go`.
- Refactored DTO conversion logic, specifically for `UserResponse` (using functional options) and `Report` responses.
- Removed redundant/deprecated DTOs like `PostDetailResponse` and `TradeItemDetailResponse`.
- **Cache Improvements**:
- Enhanced `LayeredCache` with `SetRaw` to avoid double-encoding when promoting values from Redis to local cache.
- Added `DeleteBatch` support to the cache interface.
- **Other Changes**:
- Cleaned up `config.go` by removing redundant default values and explicit environment variable overrides.
- Improved WebSocket registration flow to handle connection limits gracefully.
Add comprehensive security improvements across the application:
- **IP-based login protection**: Implement IP ban system tracking login failures, auto-banning after threshold exceeded
- **Ownership verification**: Add userID parameter to Delete/Update operations for posts and comments to prevent unauthorized modifications
- **SSRF protection**: Add URL and resolved host validation for image URLs in chat and OpenAI client
- **SQL injection prevention**: Add EscapeLikeWildcard utility to escape special characters in LIKE queries
- **HTTP security**: Configure server timeouts and add security headers middleware
- **Rate limiting**: Refactor to support configurable duration and per-endpoint rate limits for auth routes
- **Error handling**: Standardize error responses using HandleError and proper error types
- Remove BOM from all Go source files
- Standardize import ordering and whitespace across handlers and services
- Replace PostgreSQL full-text search with ILIKE pattern matching in post and user repositories
- Enhance JPush client with TLS transport, rate limit monitoring, and simplified API
- Refactor PushService to include userID in device management methods
- Add MaxDevicesPerUser limit and extract helper functions for push payload construction
Add JPush (极光推送) integration to enable push notifications for offline users. This includes:
- New jpush client package with API for batch push notifications
- Configuration for jpush (enabled, app_key, master_secret, production mode)
- Updated push service to send messages via JPush when users are offline
- Added PushChatMessage method with do-not-disturb checking
- Integrated push service with chat service to notify offline users of new messages
- Updated deployment workflow with JPush and related environment variables
Add complete trade functionality including:
- TradeItem, TradeImage, TradeFavorite models with auto-migration
- TradeRepository with CRUD and favorite operations
- TradeService with business logic for listing, creating, updating, status management
- TradeHandler with RESTful endpoints for trade operations
- DTOs and converters for request/response handling
Routes include: list, get by id, create, update, delete, status update, view recording, favorite/unfavorite.
Move the scoped 'scored' struct definition to package level and rename
to hotRankScored to enable reuse across Refresh and refreshChannelRanks
methods, eliminating duplicate type definitions.
Add channel-specific hot post filtering and ranking across the posts
listing flow. Introduce refreshChannelRanks to compute per-channel hot
scores, new Redis cache keys for channel hot ranks, and update
repository/service/handler layers to accept channelID filtering
parameters. Update HotRankWorker dependency injection to include
ChannelRepository.
Add support for referencing other posts within messages through a new post_ref segment type. Includes new PostReference model to track relationships, PostRefService for processing references, and new endpoints for post suggestions, related posts, and reference click tracking.
- Rename Muted field to NotificationMuted across model, DTOs and converters
- Add UpdateNotificationMuted repository method with upsert support
- Add SetConversationNotificationMuted service method
- Add PUT /:id/notification_muted route for setting mute status
- Improve S3 storage resilience by skipping bucket check on access denied
- Change type field to always be "notification" and add system_type field
- Rename "extra" and "data" fields to "extra_data" for consistency
- Add "is_read" field to notification payload
- Use PublishToUserOnline instead of PublishToUser to prevent system
messages from being stored in reconnection history replay
- System notifications are already persisted to database, clients can
fetch via REST API on reconnect
Add functionality to filter out posts from blocked users in post listing endpoints. Includes a new `IsBlockedBatch` method for efficient batch checking of blocked relationships and `is_blocked` field in API responses when viewing a blocked user's posts.
Remove external UI Avatars API dependency and use a static default avatar
URL instead. Simplifies avatar utility functions by eliminating dynamic
avatar generation.
Add POST /api/v1/admin/setup-super-admin endpoint to initialize the first super admin.
The endpoint can only be used once - after a super admin exists, subsequent requests
are rejected. Requires valid setup_secret configured via APP_SETUP_SECRET environment
variable or setup_secret in config file.
Add rich message segment support enabling structured content (text, images, votes, mentions) in posts and comments. Includes segments field in models, DTOs, handlers, and services. Also adds @mention notification processing and migrates vote data to embedded segments.
The Go module name has been changed from `carrot_bbs` to `with_you`, which requires
all import paths to be updated throughout the codebase and by external consumers.
BREAKING CHANGE: Module name changed from carrot_bbs to with_you. All imports
and dependencies must be updated from carrot_bbs/* to with_you/*.
Add Tencent Cloud Text Moderation System (TMS) as a fallback moderation
service when OpenAI is unavailable or returns an error. The service
now checks OpenAI first, and falls back to Tencent TMS if it fails,
providing better reliability for content moderation.
Features:
- New Tencent TMS configuration in config.yaml with environment variable overrides
- New tencent package in internal/pkg/tencent/ with TMS client implementation
- PostAIService now uses dual moderation with OpenAI primary and Tencent fallback
- Strict mode configuration passed to PostAIService for consistent behavior
BREAKING CHANGE: NewPostAIService now requires tencentClient and strictMode
parameters - update wire configuration accordingly.
Add admin comment moderation endpoints (single and batch) and introduce a three-tier moderation system (pass/review/block) for content flagged by AI. Content with unclear violations is now held for manual review instead of being auto-rejected. Deleted the legacy audit_service.go in favor of the unified hook-based moderation system.
Add PreviewURL and PreviewURLLarge fields to PostImage model and responses. Image data now supports format "url|preview_url|preview_url_large" for storing multiple image variants. Also change comment cursor pagination to sort by created_at ASC (oldest first) to display earliest comments as first floor, and make AdminReviewVerificationRequest.Approve field optional.
Add privacy settings allowing users to control visibility of their followers, following, posts, and favorites lists with three visibility levels: everyone, following, and self. Also implement account deletion workflow with 30-day cooldown period.
- Add PrivacySettings model with visibility levels
- Add DTOs for privacy settings and account deletion
- Add repository methods for privacy settings and deletion
- Add service methods to check visibility permissions
- Add handlers for get/update privacy settings and account deletion
- Add new routes for privacy settings and account management
- Add account cleanup service for hard-deleting expired accounts
- Modify login to cancel deletion when user logs in
BREAKING CHANGE: Users with pending_deletion status can no longer log in; logging in now automatically cancels deletion requests.
Implement sensitive word filtering feature with configurable database and Redis
support. Add security enhancements including WebSocket origin validation, improved
password strength requirements, timing attack prevention, and production JWT secret
validation. Add batch operation validation limits and optimize user blocking
queries with subqueries.
BREAKING CHANGE: Password validation now requires minimum 8 characters with uppercase,
lowercase, and digit (was 6-50 characters without complexity requirements)
Implement a complete user identity verification system with support for multiple identity types (student, teacher, staff, alumni). The system includes user-facing verification submission and status checking endpoints, admin endpoints for reviewing verification requests, middleware for protecting verification-required routes, and integration with the user model to track verification status.
- Replace `interface{}` with `any` type alias across all packages
- Use built-in `min()`/`max()` for parameter clamping
- Use `slices.SortFunc`, `slices.Min`, `slices.Max` for cleaner code
- Use `strings.Cut()` for simpler string parsing in auth middleware
- Use `errors.Is()` for proper error comparison in handlers
- Update dependencies: golang.org/x/image 0.37.0 -> 0.38.0
- Add Wire code generation guidelines to ARCHITECTURE.md
- Disable Go cache in CI build workflow
- Added Report and AdminReport handlers to manage report-related functionalities.
- Updated router and wire generation to include new report services and handlers.
- Enhanced error handling in report and admin report handlers to return appropriate status codes.
- Refactored notification logic in admin report service to utilize a dedicated notification repository and push service.
- Removed unused time imports from report DTO and service files.
- Updated import paths for response handling in admin and general report handlers.
- Refactored logging methods in admin and report services to use a unified operation logging approach.
- Introduced helper functions for extracting text from message segments and converting user models to brief report information.
- Introduced ActiveCall and ActiveParticipant types to manage call state in memory.
- Updated CallService interface to return ActiveCall instances for Invite, Accept, and End methods.
- Added methods for storing, retrieving, and removing active calls from memory, enhancing call handling efficiency.
- Implemented concurrency control with sync.RWMutex to ensure thread-safe access to active call data.
- Remove SetLogService methods from user, post, comment, and admin post services
- Add logService as constructor parameter to all dependent services
- Centralize call-related and message error definitions in app_errors.go
- Add ConversationID field to WSEventResponse for improved message tracking
- Simplify wire provider functions by removing manual setter calls
- Updated CallService to handle user connection and disconnection events, allowing for automatic management of active calls based on user status.
- Introduced methods to retrieve active calls for users and send pending call invites when users reconnect.
- Modified the Invite method to return the callee's online status, improving user experience during call invitations.
- Enhanced WSHandler to support connection and disconnection event handlers, ensuring accurate call state management.
- Updated the WSHandler to include a media type field in the call invite request, allowing differentiation between voice and video calls.
- Modified the CallService interface and implementation to accept the media type parameter during call invitations, enhancing call handling capabilities.
- Introduced a new method, PublishToUserOnlineReliable, to ensure critical signaling messages are reliably sent to online users.
- Updated CallService to include a cleanup mechanism for expired calls, improving resource management and user experience.
- Enhanced call acceptance logic to utilize optimistic locking for state updates, ensuring accurate call status handling across devices.
- Refactored wire generation to accommodate the new database dependency in CallService, streamlining service initialization.
- Added error handling for call invite and answer processes to differentiate between callee offline and call already answered scenarios.
- Introduced a new method to publish messages only to online users, improving real-time communication during call signaling.
- Updated call service to check callee's online status before sending call invites, ensuring better user experience.
- Enhanced call acceptance logic to notify both caller and callee about call status changes, including when a call is answered on another device.
- Added CallHandler and related services for managing call sessions and participants.
- Enhanced WSHandler to support call signaling, including invite, answer, reject, and end functionalities.
- Updated router to include call-related routes for history and ICE server retrieval.
- Introduced WebRTC configuration in the application settings for call management.
- Refactored wire generation to include CallService and CallRepository for improved dependency injection.
- Added WebSocket support by introducing a new WSHandler and related infrastructure.
- Replaced SSE implementations with WebSocket equivalents across message and QR code handlers.
- Updated service and repository layers to utilize WebSocket for real-time messaging.
- Removed obsolete SSE hub and related code, streamlining the application for WebSocket usage.
- Enhanced router to include WebSocket endpoints for real-time communication.
- Refactored repository structures to use interfaces instead of concrete types, enhancing flexibility and testability.
- Updated various repository methods to accept interfaces, allowing for better dependency management.
- Modified wire generation to accommodate new repository interfaces, ensuring proper service injection throughout the application.
- Enhanced handler methods to utilize DTOs for filtering and data handling, improving code clarity and maintainability.
