Question

We have a weird situation going on here. We have setup a local installation of NuGet gallery (the same code that powers http://Nuget.org). The IIS and the DB are on the same box. The IIS App pool has been configured to run under a domain user who is the local admin on the box as well and has the “right to logon as a service” rights. Now when someone (after succesfully logging in) tries and upload a package, the upload package business takes ages and eventually throws up the “The underlying provider failed to open” error

  [TimeoutException: Transaction Timeout]

[TransactionException: The operation is not valid for the state of the transaction.]
   System.Transactions.TransactionState.EnlistPromotableSinglePhase(InternalTransaction tx, IPromotableSinglePhaseNotification promotableSinglePhaseNotification, Transaction atomicTransaction) +53
   System.Transactions.Transaction.EnlistPromotableSinglePhase(IPromotableSinglePhaseNotification promotableSinglePhaseNotification) +241
   System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx) +273
   System.Data.ProviderBase.DbConnectionInternal.ActivateConnection(Transaction transaction) +150
   System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject) +2647
   System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection) +89
   System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory) +6372062
   System.Data.SqlClient.SqlConnection.Open() +300
   System.Data.EntityClient.EntityConnection.OpenStoreConnectionIf(Boolean openCondition, DbConnection storeConnectionToOpen, DbConnection originalConnection, String exceptionCode, String attemptedOperation, Boolean& closeStoreConnectionOnFailure) +67

[EntityException: The underlying provider failed on Open.]
   System.Data.EntityClient.EntityConnection.OpenStoreConnectionIf(Boolean openCondition, DbConnection storeConnectionToOpen, DbConnection originalConnection, String exceptionCode, String attemptedOperation, Boolean& closeStoreConnectionOnFailure) [phone]
   System.Data.EntityClient.EntityConnection.Open() +142
   System.Data.Objects.ObjectContext.EnsureConnection() +97
   System.Data.Objects.ObjectContext.ExecuteStoreQueryInternal(String commandText, String entitySetName, MergeOption mergeOption, Object[] parameters) +109
   System.Data.Objects.ObjectContext.ExecuteStoreQuery(String commandText, Object[] parameters) +87
   System.Data.Entity.Internal.InternalContext.ExecuteSqlQuery(String sql, Object[] parameters) +118
   System.Data.Entity.Internal.InternalContext.ExecuteSqlQueryAsIEnumerable(String sql, Object[] parameters) +85
   System.Data.Entity.Internal.InternalContext.ExecuteSqlQuery(Type elementType, String sql, Object[] parameters) +241
   System.Data.Entity.Internal.InternalSqlNonSetQuery.GetEnumerator() +34
   System.Data.Entity.Internal.InternalSqlQuery`1.GetEnumerator() +28
   System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) +382
   System.Linq.Enumerable.ToList(IEnumerable`1 source) +80
   NuGetGallery.LuceneIndexingService.GetPackages(DbContext context, Nullable`1 dateTime) in C:\Source\DotNetDevEng.CitiNuGet\Gallery\DEV\Website\Infrastructure\Lucene\LuceneIndexingService.cs:55
   NuGetGallery.LuceneIndexingService.UpdateIndex() in C:\Source\DotNetDevEng.CitiNuGet\Gallery\DEV\Website\Infrastructure\Lucene\LuceneIndexingService.cs:32
   NuGetGallery.PackageService.CreatePackage(IPackage nugetPackage, User currentUser) in C:\Source\DotNetDevEng.CitiNuGet\Gallery\DEV\Website\Services\PackageService.cs:61
   NuGetGallery.PackagesController.VerifyPackage(Nullable`1 listed) in C:\Source\DotNetDevEng.CitiNuGet\Gallery\DEV\Website\Controllers\PackagesController.cs:503
   lambda_method(Closure , ControllerBase , Object[] ) +121
   System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters) +248
   System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters) +39
   System.Web.Mvc.<>c__DisplayClass15.<InvokeActionMethodWithFilters>b__12() +125
   System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func`1 continuation) +640
   System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func`1 continuation) +640
   System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodWithFilters(ControllerContext controllerContext, IList`1 filters, ActionDescriptor actionDescriptor, IDictionary`2 parameters) +312
   System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName) +691
   System.Web.Mvc.Controller.ExecuteCore() +162
   System.Web.Mvc.ControllerBase.Execute(RequestContext requestContext) +305
   System.Web.Mvc.<>c__DisplayClassb.<BeginProcessRequest>b__5() +62
   System.Web.Mvc.Async.<>c__DisplayClass1.<MakeVoidDelegate>b__0() +20
   System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +469
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +375

Interestingly if I switch IIS App Pool to run under the built in “LocalSystem” account all these problems goes away. The domain user we want to use has been temporarily setup as the database owner for NuGetGallery database but it isn’t helping.

Can anyone suggest what setting/configuration we might be missing here?

Answer 1

This was due to ambient transactions going on within the code. The code relied on TransactionScope classes to manage various transactions which were getting in a jam when they got nested. Things turned particularly bad if someone tried uploading a package which was more than around 10MB in size. Once we tweaked the MSDTC on the boxes the application started behaving slightly properly. However it was still intermittently getting deadlocked within its own transactions so for now we have decided to remove transactions from the package upload bit.

For anyone else who gets into this problem I would recommend either of these two

1. Comment out any reference to TransactionScope in

Services\PackageService.cs (method CreatePackage, line 48)

  using (var tx = new TransactionScope())
            {
                using (var stream = nugetPackage.GetStream())
                {
                    UpdateIsLatest(packageRegistration);
                    packageRegistrationRepo.CommitChanges();
                    packageFileSvc.SavePackageFile(package, stream);
                    tx.Complete();
                }
            }

And in Controller\PackageController.cs method VerifyPackage(bool? Listed) line 501

 using (var tx = new TransactionScope())
            {

                package = packageSvc.CreatePackage(nugetPackage, currentUser);
                packageSvc.PublishPackage(package.PackageRegistration.Id, package.Version);
                if (listed.HasValue && listed.Value == false)
                    packageSvc.MarkPackageUnlisted(package);
                uploadFileSvc.DeleteUploadFile(currentUser.Key);
                tx.Complete();
            }

2 Uncomment those lines and follow the approach mentioned here http://blogs.msdn.com/b/dbrowne/archive/2010/05/21/using-new-transactionscope-considered-harmful.aspx

Basically create a transactionscope which can work well with SQL Server. Create a new class in the InfraStructureFolder, call it TranscationUtils and copy the code in there. Then the using blocks would look like

using (var tx = new TransactionUtils.CreateTransactionScope()){

//cod here
tx.Complete();
}

However this second approach can result in another server error which again relates to the default time outs you can specify for TransactionScope. We have personally decided to go with approach 1 and comment out all TransactionScope references for now.